Re: [Crash-utility] crash for live system fails to load - Ubuntu 16.10

Hi, I've been using crash for a while.. Strangely, on Ubuntu (and Lubuntu) 16.10, I face the issue described below: $ cat /etc/issue Ubuntu 16.10 \n \l $ uname -r 4.8.0-46-generic $ cat /proc/version Linux version 4.8.0-46-generic (buildd@lcy01-16) (gcc version 6.2.0 20161005 (Ubuntu 6.2.0-5ubuntu12) ) #49-Ubuntu SMP Fri Mar 31 13:57:14 UTC 2017 $ cat /proc/version_signature Ubuntu 4.8.0-46.49-generic 4.8.17 $ So I download the dbgsym package for this kernel as: $ wget http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-4.8.0-46-generic-db... ... Then extract the vmlinux image from the package. $ ls -l /usr/lib/debug/boot/vmlinux-4.8.0-46-generic -rw-r--r-- 1 root root 509M Mar 31 12:43 /usr/lib/debug/boot/vmlinux-4.8.0-46-generic $ Run crash for 'live system': $ sudo crash /usr/lib/debug/boot/vmlinux-4.8.0-46-generic /proc/kcore /boot/System.map-4.8.0-46-generic [sudo] password for kaiwan: crash 7.1.5 Copyright (C) 2002-2016 Red Hat, Inc. This GDB was configured as "x86_64-unknown-linux-gnu"... --snip-- crash: seek error: kernel virtual address: ffffffff81f5dcc0 type: "possible" WARNING: cannot read cpu_possible_map crash: seek error: kernel virtual address: ffffffff81f5dc40 type: "present" WARNING: cannot read cpu_present_map crash: seek error: kernel virtual address: ffffffff81f5dc80 type: "online" WARNING: cannot read cpu_online_map crash: seek error: kernel virtual address: ffffffff81f5dc00 type: "active" WARNING: cannot read cpu_active_map crash: seek error: kernel virtual address: ffffffff81e2a070 type: "pv_init_ops" crash: seek error: kernel virtual address: ffffffff82167130 type: "shadow_timekeeper xtime_sec" crash: seek error: kernel virtual address: ffffffff81e0d304 type: "init_uts_ns" crash: /boot/System.map-4.8.0-46-generic and /proc/kcore do not match! Usage: crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form) crash [OPTION]... [NAMELIST] (live system form) Enter "crash -h" for details. $ It _does_ load up with the --minimal switch, but of course I'd like to run it with full capabilities. FYI, output with debug ON: $ sudo crash -d5 /usr/lib/debug/boot/vmlinux-4.8.0-46-generic /proc/kcore /boot/System.map-4.8.0-46-generic crash 7.1.5 Copyright (C) 2002-2016 Red Hat, Inc. --snip-- get_live_memory_source: /proc/kcore proc_kcore_data: flags: 0 (KCORE_LOCAL|KCORE_ELF64) segments: 17 elf_header: 55f59ded06e0 elf64: 55f59ded06e0 load64: 55f59ded0758 elf32: 0 load32: 0 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 7fffff602000 p_vaddr: ffffffffff600000 p_paddr: 0 p_filesz: 1000 p_memsz: 1000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 7fff9da02000 p_vaddr: ffffffff9da00000 p_paddr: 0 p_filesz: 1268000 p_memsz: 1268000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 2922c0002000 p_vaddr: ffffa922c0000000 p_paddr: 0 p_filesz: 1fffffffffff p_memsz: 1fffffffffff p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 7fffc0002000 p_vaddr: ffffffffc0000000 p_paddr: 0 p_filesz: 3f000000 p_memsz: 3f000000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 16a100003000 p_vaddr: ffff96a100001000 p_paddr: 0 p_filesz: 57000 p_memsz: 57000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 6a0000002000 p_vaddr: ffffea0000000000 p_paddr: 0 p_filesz: 2000 p_memsz: 2000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 16a10005b000 p_vaddr: ffff96a100059000 p_paddr: 0 p_filesz: 33000 p_memsz: 33000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 16a100102000 p_vaddr: ffff96a100100000 p_paddr: 0 p_filesz: 77b5f000 p_memsz: 77b5f000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 6a0000006000 p_vaddr: ffffea0000004000 p_paddr: 0 p_filesz: 1dee000 p_memsz: 1dee000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 16a177c63000 p_vaddr: ffff96a177c61000 p_paddr: 0 p_filesz: 2fb000 p_memsz: 2fb000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 16a177f5f000 p_vaddr: ffff96a177f5d000 p_paddr: 0 p_filesz: 34af4000 p_memsz: 34af4000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 6a0001dff000 p_vaddr: ffffea0001dfd000 p_paddr: 0 p_filesz: d2d000 p_memsz: d2d000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 16a1aca7e000 p_vaddr: ffff96a1aca7c000 p_paddr: 0 p_filesz: 9903000 p_memsz: 9903000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 16a1b8001000 p_vaddr: ffff96a1b7fff000 p_paddr: 0 p_filesz: 1000 p_memsz: 1000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 6a0002e01000 p_vaddr: ffffea0002dff000 p_paddr: 0 p_filesz: 1000 p_memsz: 1000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 16a200002000 p_vaddr: ffff96a200000000 p_paddr: 0 p_filesz: 142800000 p_memsz: 142800000 p_align: 4096 Elf64_Phdr: p_type: 1 p_flags: 7 p_offset: 6a0004002000 p_vaddr: ffffea0004000000 p_paddr: 0 p_filesz: 50a0000 p_memsz: 50a0000 p_align: 4096 readmem: read_proc_kcore() -> /proc/kcore crash: pv_init_ops exists: ARCH_PVOPS _text: ffffffff81000000 Kernel code: 1aaa00000 -> phys_base: 1a9a00000 gdb /usr/lib/debug/boot/vmlinux-4.8.0-46-generic GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... <readmem: ffffffff81f5dcc0, KVADDR, "possible", 64, (ROE), 55f59c515060> <read_proc_kcore: addr: ffffffff81f5dcc0 paddr: 1ab95dcc0 cnt: 64> crash: seek error: kernel virtual address: ffffffff81f5dcc0 type: "possible" WARNING: cannot read cpu_possible_map <readmem: ffffffff81f5dc40, KVADDR, "present", 64, (ROE), 55f59c515060> <read_proc_kcore: addr: ffffffff81f5dc40 paddr: 1ab95dc40 cnt: 64> crash: seek error: kernel virtual address: ffffffff81f5dc40 type: "present" WARNING: cannot read cpu_present_map <readmem: ffffffff81f5dc80, KVADDR, "online", 64, (ROE), 55f59c515060> <read_proc_kcore: addr: ffffffff81f5dc80 paddr: 1ab95dc80 cnt: 64> crash: seek error: kernel virtual address: ffffffff81f5dc80 type: "online" WARNING: cannot read cpu_online_map <readmem: ffffffff81f5dc00, KVADDR, "active", 64, (ROE), 55f59c515060> <read_proc_kcore: addr: ffffffff81f5dc00 paddr: 1ab95dc00 cnt: 64> crash: seek error: kernel virtual address: ffffffff81f5dc00 type: "active" WARNING: cannot read cpu_active_map <readmem: ffffffff81e2a070, KVADDR, "pv_init_ops", 8, (ROE), 7fff3eafb4b0> <read_proc_kcore: addr: ffffffff81e2a070 paddr: 1ab82a070 cnt: 8> crash: seek error: kernel virtual address: ffffffff81e2a070 type: "pv_init_ops" <readmem: ffffffff82167130, KVADDR, "shadow_timekeeper xtime_sec", 8, (ROE), 7fff3eafb460> <read_proc_kcore: addr: ffffffff82167130 paddr: 1abb67130 cnt: 8> crash: seek error: kernel virtual address: ffffffff82167130 type: "shadow_timekeeper xtime_sec" xtime timespec.tv_sec: 8: Thu Jan 1 05:30:08 1970 <readmem: ffffffff81e0d304, KVADDR, "init_uts_ns", 390, (ROE), 55f59c2b9e3c> <read_proc_kcore: addr: ffffffff81e0d304 paddr: 1ab80d304 cnt: 390> crash: seek error: kernel virtual address: ffffffff81e0d304 type: "init_uts_ns" utsname: sysname: nodename: release: version: machine: domainname: base kernel version: 0.0.0 <readmem: ffffffff81a000c0, KVADDR, "accessible check", 8, (ROE|Q), 7fff3eaf87b0> <read_proc_kcore: addr: ffffffff81a000c0 paddr: 1ab4000c0 cnt: 8> crash: seek error: kernel virtual address: ffffffff81a000c0 type: "accessible check" crash: /boot/System.map-4.8.0-46-generic and /proc/kcore do not match! Usage: crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form) crash [OPTION]... [NAMELIST] (live system form) Enter "crash -h" for details. $ The similar procedure worked just fine on [L]Ubuntu 15.10. Any help in solving this would be appreciated, TIA! -Kaiwan. Thanks & Regards, Kaiwan.