Download from: http://people.redhat.com/anderson or https://github.com/crash-utility/crash/releases The github master branch serves as a development branch that will contain all patches that are queued for the next release: $ git clone git://github.com/crash-utility/crash.git Changelog: - Fix for the "timer -r" command on Linux 4.10 and later kernels that contain commit 2456e855354415bfaeb7badaa14e11b3e02c8466, titled "ktime: Get rid of the union". Without the patch, the command fails with the error message "timer: invalid structure member offset: ktime_t_sec". (k-hagio(a)ab.jp.nec.com) - Fix for the x86 and x86_64 "mach -m" option on Linux 4.12 and later kernels to account for the structure name changes "e820map" to "e820_table", and "e820entry" to "e820_entry", and for the symbol name change from "e820" to "e820_table". Also updated the display output to properly translate E820_PRAM and E820_RESERVED_KERN entries. Without the patch on all kernels, E820_PRAM and E820_RESERVED_KERN entries show "type 12" and "type 128" respectively. Without the patch on Linux 4.12 and later kernels, the command fails with the error message "mach: cannot resolve e820". (anderson(a)redhat.com) - Update for the recognition of the new x86_64 CPU_ENTRY_AREA virtual address range introduced in Linux 4.15. The memory range exists above the vmemmap range and below the mapped kernel static text/data region, and where all of the x86_64 exception stacks have been moved. Without the patch, reads from the new memory region fail because the address range is not recognized as a legitimate virtual address. Most notable is the failure of "bt" on tasks whose backtraces originate from any of the exception stacks, which fail with the two error messages "bt: seek error: kernel virtual address: <address> type: stack contents" followed by "bt: read of stack at <address> failed". (anderson(a)redhat.com) - Fix to address a "__builtin___snprintf_chk" compiler warning if bpf.c is compiled with -D_FORTIFY_SOURCE=2. (anderson(a)redhat.com) - Fix for the "bpf -t" option. Although highly unlikely, without the patch, the target function name of a BPF bytecode call instruction may fail to be resolved correctly. (anderson(a)redhat.com) - If /proc/kcore gets selected for the live memory source because /dev/mem was configured with CONFIG_STRICT_DEVMEM, its ELF header contents are not displayed by "help -[dD]", and are not displayed when the crash session is invoked with -d<number>". Without the patch, the ELF contents are only displayed in those two situations if "/proc/kcore" is explicitly entered on the crash command line. (anderson(a)redhat.com) - If the default live memory source /dev/mem is determined to be unusable because the kernel was configured with CONFIG_STRICT_DEVMEM, the first memory read during session initialization will fail. The current behavior results in a readmem() error message, followed by two notification messages that indicate that /dev/mem is restricted and a switch to using /proc/kcore will be attempted; the readmem is reattempted from /proc/kcore, and if successful, the session will continue initialization. With this patch, the behavior will change such that if the switch to /proc/kcore and the reattempted readmem() are successful, no messages will be displayed unless the crash session is invoked with "crash -d<number>". (anderson(a)redhat.com) - Fix for the ppc64/ppc64le "bt" command on Linux 4.7 and later kernels that contain commit d8bff643d81a58181356c0aa3ab771ac10da6894, titled "[x86] asm: Make sure verify_cpu() has a good stack", which inadvertently breaks the ppc64/ppc64le kernel stack size calculation when running with crash-7.2.2 or later. Without the patch, "bt" may fail with a filtered kdump dumpfile with the two error messages "bt: page excluded: kernel virtual address: <address> type: stack contents" and "bt: read of stack at <address> failed". (anderson(a)redhat.com) - Fix for PPC64 kernel virtual address translation in Linux 4.17 and later kernels with commit c2b4d8b7417a59b7f9a52d0d8402f5257cbbd398, titled "powerpc/mm/hash64: Increase the VA range", in which the maximum virtual address value has been increased to 4PB. Without the patch, the translation/access of high vmalloc space addresses fails; for example, the "kmem -[sS]" option fails the translation of per-cpu kmem_cache_cpu addresses located in vmalloc space, with the error messages "kmem: invalid kernel virtual address: <address> type: kmem_cache_cpu.freelist" and "kmem: invalid kernel virtual address: <address> type: kmem_cache_cpu.page", and the "vtop" command shows the addresses as "(not mapped)". (hbathini(a)linux.ibm.com) - Fix for the x86_64 "bt" command in which a legitimate exception frame is appended with the message "bt: WARNING: possibly bogus exception frame". This only happens in KASLR-enabled kernels when the text address that was executing when the exception occurred is marked as a "weak" symbol (type "W") instead of a text symbol (type "T" or "t"). As a result, the exception frame's RIP is not recognized as a text symbol, and the warning message is displayed. (anderson(a)redhat.com) - Fix for the x86_64 "bt" command in Linux 4.16 and later kernels containing commit 3aa99fc3e708b9cd9b4cfe2df0b7a66cf293e3cf, titled "x86/entry/64: Remove 'interrupt' macro". Without the patch, the exception frame display generated by an interrupt exception will show incorrect contents, and be followed by the message "bt: WARNING: possibly bogus exception frame". (anderson(a)redhat.com) - Fix for the failure of several "kmem" command options, most notably seen if the command is piped directly into a crash session, or if the command is contained in an input file. For examples: $ echo "kmem -i" | crash ... $ crash -i <input-file> ... Without the patch, the kmem command may fail with the error message "<segmentation violation in gdb>". While the bug is due to a buffer overflow that has always existed, it only is triggered by certain kernel configurations. (anderson(a)redhat.com) - Update for the "kmem -V" option to also dump the global entries that are contained in the "vm_numa_stat" array that was introduced in Linux 4.14. Also, the command output separates the "vm_zone_stat", "vm_node_stat" and "vm_numa_stat" entries into separate sections with "VM_ZONE_STAT", "VM_NODE_STAT" and "VM_NUMA_STAT" headers. Without the patch, the "vm_zone_stat" and "vm_node_stat" entries are listed together under a "VM_STAT" header. (anderson(a)redhat.com) - Support for the "bpf" command on RHEL 3.10.0-913.el7 and later 3.10-based RHEL7 kernels, which contain a backport of the upstream eBPF code, but still use the older, pre-4.11, IDR facility that does not use radix trees for linking the active bpf_prog and bpf_map structures. Without the patch, the command indicates "bpf: command not supported or applicable on this architecture or kernel". (anderson(a)redhat.com) - Third phase of support for x86_64 5-level page tables in Linux 4.17 and later kernels. With this patch, the usage of 5-level page tables is automatically detected on live systems and when running against vmcores that contain the new "NUMBER(pgtable_l5_enabled)" VMCOREINFO entry. Without the patch, the "--machdep vm=5level" command line option is required. (douly.fnst(a)cn.fujitsu.com, anderson(a)redhat.com) - The existing "list" command uses a hash table to detect duplicate items as it traverses the list. The hash table approach has worked well for many years. However, with increasing memory sizes and list sizes, the overhead of the hash table can be substantial, often leading to commands running for a very long time. For large lists, we have found that the existing hash based approach may slow the system to a crawl and possibly never complete. You can turn off the hash with "set hash off" but then there is no loop detection; in that case, loop detection must be done manually after dumping the list to disk or some other method. This patch is an implementation of the cycle detection algorithm from R. P. Brent as an alternative algorithm for the "list" command. The algorithm both avoids the overhead of the hash table and yet is able to detect a loop. In addition, further loop characteristics are printed, such as the distance to the start of the loop as well as the loop length. An excellent description of the algorithm can be found here on the crash-utility mailing list: https://www.redhat.com/archives/crash-utility/2018-July/msg00019.html A new "list -B" option has been added to the "list" command to invoke this new algorithm rather than using the hash table. In addition to low memory usage, the output of the list command is slightly different when a loop is detected. In addition to printing the first duplicate entry, the length of the loop, and the distance to the loop is output. (dwysocha(a)redhat.com) - Fix for x86_64 "bt" command to prevent an in-kernel exception frame from not being displayed. Without the patch, if the RIP in a pt_regs structure on the stack is not a kernel text address, such as a NULL pointer, it is not recognized as an exception frame and the register set is not displayed. (anderson(a)redhat.com) - Fix for the "repeat" command when the argument consists of an input file construct, for example, "repeat -1 < input_file". Without the patch, only the first command line in the input file is executed each time. (anderson(a)redhat.com) - Fourth phase of support for x86_64 5-level page tables in Linux 4.17 and later kernels. This patch adds support for user virtual address translation when the kernel is configured with CONFIG_X86_5LEVEL. (douly.fnst(a)cn.fujitsu.com) - Fix to prevent an unnecessary "read error" message during session initialization on live systems running a kernel that is configured with CONFIG_X86_5LEVEL. Without the patch, a message indicating "crash: read error: kernel virtual address: <address> type: __pgtable_l5_enabled" will be displayed if /proc/kcore gets selected as the live memory source after /dev/mem is determined to be unusable. (anderson(a)redhat.com) - Update for "ps" and "foreach" commands to display and recognize two new process states, "ID" for the TASK_IDLE macro introduced in Linux 4.2, and "NE" for the TASK_NEW bit introduced in Linux 4.8. (k-hagio(a)ab.jp.nec.com) - Fix for running live on ARM64 kernels against /proc/kcore on kernels configured with CONFIG_RANDOMIZE_BASE. Without the patch, depending upon the hardware platform, the session may fail with the error message "crash: vmlinux and /proc/kcore do not match!". (anderson(a)redhat.com) - Modify the output of the "kmem -[sS]" header and contents such that the slab cache "NAME" string is moved from the second column to the the last column. Since the slab cache name strings have become increasingly longer over time, without the patch, the numerical column contents may be skewed so far to the right that the output becomes difficult to read. (k-hagio(a)ab.jp.nec.com) - Fix for the "files" and "net -s" commands when a task has an open files count that exceeds 1024 (FD_SETSIZE) file descriptors. Without the patch, the commands may omit the display of open file descriptors. (tan.hu(a)zte.com.cn) - As an addendum to the new "kmem -[sS]" output format, align the slab cache name string so that it is beneath the "NAME" header column when the "kmem -I <slab-cache>" option is used to ignore a slab cache, or if the scan of the metadata of a slab cache enounters corruption. Also remove a superfluous line from the "help kmem" description of the "kmem -I" option. (k-hagio(a)ab.jp.nec.com, anderson(a)redhat.com) - Account for the addition of the new ORC unwinder "orc_entry.end" member in kernel commit d31a580266eeb1f355df90fde8a71f480e30ad70, titled "x86/unwind/orc: Detect the end of the stack". (anderson(a)redhat.com) - Fix for the "trace.c" extension module for RHEL7.6, which moved the ftrace_event_call.data member into a new structure contained within an anonymous union. Without the patch, the module fails to load, indicating "no commands registered: shared object unloaded". (xuhuan.fnst(a)cn.fujitsu.com) - Fix for the "vm -p", user-space "vtop", and "pte" commands in kernels where the dimension of the static swap_info[] array is not contained in the vmlinux file's debuginfo data. Without the patch, the translation of a swapped-out PTE entry fails to determine the swap device, and the commands display "cannot determine swap location". (anderson(a)redhat.com) - Fix for the swap offset calculation in the x86_64 "vm -p", "pte", and user-space "vtop" commands. The swap offset bits in an x86_64 PTE were changed in Linux 4.6, and then again in Linux 4.18.1 with the new L1TF security patchset. Without the patch, the offset value in the later kernels, or in older kernels with an L1TF backport, show an incorrect swap offset value. (anderson(a)redhat.com) - Fix for the "kmem -V" option on Linux 4.14 and later kernels that are configured without CONFIG_NUMA, and therefore do not contain the "numa_stat_item" enumeration. Without the patch, the command causes the crash session to abort with the error messages "double free or corruption (!prev)" followed by "Aborted (core dumped)". (k-hagio(a)ab.jp.nec.com) - Introduction of a new "kmem -r" option. With the implementation of per-cgroup kmem_cache slabs, the number of slab caches displayed by "kmem -s" can number into the thousands. Similar to /proc/slabinfo, this new option displays the accumulated data of the root cache and its children. It is limited to Linux 4.11 and later kernels that contain the "slab_root_caches" list. Currently the command option is restricted to kernels configured with CONFIG_SLUB. (k-hagio(a)ab.jp.nec.com) - Fix for Linux 4.19-rc1 and later kernels that contain kernel commit 2c4704756cab7cfa031ada4dab361562f0e357c0, titled "pids: Move the pgrp and session pid pointers from task_struct to signal_struct". Without the patch, the crash session fails during initialization with the message "crash: invalid structure member offset: task_struct_pids". (anderson(a)redhat.com) - Fix for Linux 4.19-rc1 and later kernels that contain kernel commit 7290d58095712a89f845e1bca05334796dd49ed2, titled "module: use relative references for __ksymtab entries". Without the patch, kernels configured with CONFIG_HAVE_ARCH_PREL32_RELOCATIONS fail during session initialization, with a dump of the internel buffer allocation stats followed by the message "crash: cannot allocate any more memory!" (asmadeus(a)codewreck.org) - Fix a cut-and-paste error in the previous patch application. (anderson(a)redhat.com) - Fix for the "files" command in Linux 4.17 and later kernels that contain commit b93b016313b3ba8003c3b8bb71f569af91f19fc7, titled "page cache: use xa_lock". Without the patch, the "files -c" option fails with the message "files: -c option not supported or applicable on this architecture or kernel", and the "files -p <inode>" option fails in a similar manner. (k-hagio(a)ab.jp.nec.com) - Fix for the "files -p <inode>" option. Without the patch, the command attempts to translate radix tree node slot entries that are RADIX_TREE_EXCEPTIONAL_ENTRY types, and as a result may fail prematurely with an error message of the sort "files: do_radix_tree: callback operation failed: entry: 5 item: 44788c5000a". (anderson(a)redhat.com) - Commit 3db3d3992d781c1e42587d2d2bf81e785408e0c2 in crash-7.1.8 was aimed at making the PPC64 "bt" command work for dumpfiles saved with the FADUMP facility, but it introduced a bit of unwarranted complexity in "bt" command processing. Reworked the "bt" command processing for PPC64 arch to make it a little less compilated and also to print symbols for NIP and LR registers in exception frames. Without the patch, "bt" on non-panic active tasks may fail with the message "bt: invalid kernel virtual address: <address> type: Regs NIP value". (hbathini(a)linux.ibm.com) - An addendum to crash commit 5fe78861ea1589084f6a2956a6ff63677c9269e1, this patch for the PPC64 "bt" command prevents an invalid error message from being displayed when an active non-panic task is interrupted while running in user space. Without the patch, the command correctly indicates "Task is running in user space", dumps the user-space exception frame, but then prints the invalid error message "bt: invalid kernel virtual address: ffffffffffffff90 type: Regs NIP value". (anderson(a)redhat.com)