[Crash-utility] ARM64 (odroid-c2) crash fails to read live kernel

I have the new Odroid-C2 arm64 cortex-a53 board and have been trying to get crash to work against the live kernel. I think the key error is this: linux_banner: crash: /lib/modules/3.14.29+/build/vmlinux and /dev/mem do not match! They should match as I built the kernel myself and verified the vmlinux in /lib/modules is the one I'm booted on. What concerns me is that it does not appear to be able to read anything from the vmlinux file: <read_dev_mem: addr: ffffffc001c0dbac paddr: 2c0dbac cnt: 390utsname: sysname: (not printable) nodename: release: J version: (not printable) machine: r domainname: base kernel version: 0.1.4 If I elfdump or objdump the vmlinux and grep banner I can see the symbol: root@odroid64-pre:~/linux# readelf --syms vmlinux | grep banner 74463: ffffffc00186a090 149 OBJECT GLOBAL DEFAULT 4 linux_banner 75496: ffffffc00186a028 100 OBJECT GLOBAL DEFAULT 4 linux_proc_banner root@odroid64-pre:~/linux# eu-nm -a vmlinux | grep banner linux_banner |ffffffc00186a090|GLOBAL|OBJECT |0000000000000095| version.c:43|.rodata linux_proc_banner |ffffffc00186a028|GLOBAL|OBJECT |0000000000000064| version.c:47|.rodata I pulled the crash source and built it native on the arm64 box. If I could get a pointer on where to start with debugging this it would help (i.e. which error to focus on first) === The full dump of crash startup is below: root@odroid64-pre:~/linux# /root/crash-7.1.4/crash -d 64 crash 7.1.4 Copyright (C) 2002-2015 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. find_booted_kernel: search for [Linux version 3.14.29+ (root@odroid64-pre) (gcc version 5.3.1 20160225 (Ubuntu/Linaro 5.3.1-10ubuntu2) ) #1 SMP PREEMPT Tue Mar 8 01:06:35 CST 2016] searchdirs[8]: /usr/lib/debug/lib/modules/3.14.29+/ searchdirs[0]: /usr/src/linux/ searchdirs[1]: /boot/ searchdirs[2]: /boot/efi/redhat searchdirs[3]: /boot/efi/EFI/redhat searchdirs[4]: / searchdirs[5]: /lib/modules/3.14.29+/build/ searchdirs[6]: /usr/src/redhat/BUILD/kernel-3.14.29/linux/ searchdirs[7]: /usr/src/redhat/BUILD/kernel-3.14.29/linux-3.14.29/ mount_points[0]: / (c46630) mount_points[1]: /sys (c46650) mount_points[2]: /proc (c46670) mount_points[3]: /dev (c46690) mount_points[4]: /dev/pts (c466b0) mount_points[5]: /run (c466d0) mount_points[6]: / (c466f0) mount_points[7]: /sys/kernel/security (c46710) mount_points[8]: /dev/shm (c46740) mount_points[9]: /run/lock (c46760) mount_points[10]: /sys/fs/cgroup (c46780) mount_points[11]: /sys/fs/cgroup/systemd (c467b0) mount_points[12]: /sys/fs/cgroup/devices (c467f0) mount_points[13]: /sys/fs/cgroup/cpuset (c46830) mount_points[14]: /sys/fs/cgroup/cpu,cpuacct (c46870) mount_points[15]: /sys/fs/cgroup/blkio (c468b0) mount_points[16]: /sys/fs/cgroup/debug (c468e0) mount_points[17]: /sys/fs/cgroup/perf_event (c46910) mount_points[18]: /sys/fs/cgroup/freezer (c46950) mount_points[19]: /sys/fs/cgroup/net_cls (c46990) mount_points[20]: /proc/sys/fs/binfmt_misc (c469d0) mount_points[21]: /dev/mqueue (c46a10) mount_points[22]: /sys/kernel/debug (c46a30) mount_points[23]: /dev/hugepages (c46a60) mount_points[24]: /run/rpc_pipefs (c46a90) mount_points[25]: /sys/kernel/config (c46ac0) mount_points[26]: /media/boot (c46af0) mount_points[27]: /run/cgmanager/fs (c46b10) mount_points[28]: /run/user/118 (c46b40) mount_points[29]: /run/user/118/gvfs (c46b70) mount_points[30]: /sys/fs/fuse/connections (c46ba0) mount_points[31]: /run/user/0 (c46be0) find_booted_kernel: check: /lib/modules/3.14.29+/build/vmlinux find_booted_kernel: found: /lib/modules/3.14.29+/build/vmlinux get_live_memory_source: /dev/mem /proc/version: Linux version 3.14.29+ (root@odroid64-pre) (gcc version 5.3.1 20160225 (Ubuntu/Linaro 5.3.1-10ubuntu2) ) #1 SMP PREEMPT Tue Mar 8 01:06:35 CST 2016 /lib/modules/3.14.29+/build/vmlinux: Linux version 3.14.29+ (root@odroid64-pre) (gcc version 5.3.1 20160225 (Ubuntu/Linaro 5.3.1-10ubuntu2) ) #1 SMP PREEMPT Tue Mar 8 01:06:35 CST 2016 readmem: read_dev_mem() -> /dev/mem VA_BITS: 39 using 1000000 as phys_offset gdb /lib/modules/3.14.29+/build/vmlinux GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "aarch64-unknown-linux-gnu"... GETBUF(248 -> 0) GETBUF(1500 -> 1) FREEBUF(1) FREEBUF(0) <readmem: ffffffc001874510, KVADDR, "kernel_config_data", 32768, (ROE), 17c47c0<read_dev_mem: addr: ffffffc001874510 paddr: 2874510 cnt: 2800<read_dev_mem: addr: ffffffc001875000 paddr: 2875000 cnt: 4096<read_dev_mem: addr: ffffffc001876000 paddr: 2876000 cnt: 4096<read_dev_mem: addr: ffffffc001877000 paddr: 2877000 cnt: 4096<read_dev_mem: addr: ffffffc001878000 paddr: 2878000 cnt: 4096<read_dev_mem: addr: ffffffc001879000 paddr: 2879000 cnt: 4096<read_dev_mem: addr: ffffffc00187a000 paddr: 287a000 cnt: 4096<read_dev_mem: addr: ffffffc00187b000 paddr: 287b000 cnt: 4096<read_dev_mem: addr: ffffffc00187c000 paddr: 287c000 cnt: 1296WARNING: could not find MAGIC_START! GETBUF(248 -> 0) FREEBUF(0) GETBUF(8 -> 0) <readmem: ffffffc00186fd80, KVADDR, "cpu_possible_mask", 8, (FOE), 7ffe1dfbd0<read_dev_mem: addr: ffffffc00186fd80 paddr: 286fd80 cnt: 8<readmem: 1600000102, KVADDR, "possible", 8, (ROE), bf8ae8crash: invalid kernel virtual address: 1600000102 type: "possible" WARNING: cannot read cpu_possible_map <readmem: ffffffc00186fd70, KVADDR, "cpu_present_mask", 8, (FOE), 7ffe1dfbd0<read_dev_mem: addr: ffffffc00186fd70 paddr: 286fd70 cnt: 8<readmem: 189a, KVADDR, "present", 8, (ROE), bf8ae8crash: invalid kernel virtual address: 189a type: "present" WARNING: cannot read cpu_present_map <readmem: ffffffc00186fd78, KVADDR, "cpu_online_mask", 8, (FOE), 7ffe1dfbd0<read_dev_mem: addr: ffffffc00186fd78 paddr: 286fd78 cnt: 8<readmem: 13a48, KVADDR, "online", 8, (ROE), bf8ae8crash: invalid kernel virtual address: 13a48 type: "online" WARNING: cannot read cpu_online_map <readmem: ffffffc00186fd68, KVADDR, "cpu_active_mask", 8, (FOE), 7ffe1dfbd0<read_dev_mem: addr: ffffffc00186fd68 paddr: 286fd68 cnt: 8<readmem: 1600000102, KVADDR, "active", 8, (ROE), bf8ae8crash: invalid kernel virtual address: 1600000102 type: "active" WARNING: cannot read cpu_active_map FREEBUF(0) GETBUF(248 -> 0) FREEBUF(0) GETBUF(248 -> 0) FREEBUF(0) <readmem: ffffffc001d61238, KVADDR, "timekeeper xtime_sec", 8, (ROE), 7ffe1dfc98<read_dev_mem: addr: ffffffc001d61238 paddr: 2d61238 cnt: 8xtime timespec.tv_sec: 5f044158000e9068: (null) <readmem: ffffffc001c0dbac, KVADDR, "init_uts_ns", 390, (ROE), b9606c<read_dev_mem: addr: ffffffc001c0dbac paddr: 2c0dbac cnt: 390utsname: sysname: (not printable) nodename: release: J version: (not printable) machine: r domainname: base kernel version: 0.1.4 <readmem: ffffffc00186a090, KVADDR, "accessible check", 8, (ROE|Q), 7ffe1df350<read_dev_mem: addr: ffffffc00186a090 paddr: 286a090 cnt: 8<readmem: ffffffc00186a090, KVADDR, "read_string characters", 1499, (ROE|Q), 7ffe1df6c8<read_dev_mem: addr: ffffffc00186a090 paddr: 286a090 cnt: 1499/proc/version: Linux version 3.14.29+ (root@odroid64-pre) (gcc version 5.3.1 20160225 (Ubuntu/Linaro 5.3.1-10ubuntu2) ) #1 SMP PREEMPT Tue Mar 8 01:06:35 CST 2016 linux_banner: crash: /lib/modules/3.14.29+/build/vmlinux and /dev/mem do not match! Usage: crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form) crash [OPTION]... [NAMELIST] (live system form) Enter "crash -h" for details.