> Hi Dave, >   Because I didn't subscribe the list, so please CC me when reply:) My > colleague help to forward your reply to me, so I decide to reply > directly in order to keep this mail in the thread other than new a thread. > > On 11/23/2017 02:04 PM, Fei, Jie/费 杰 wrote: >> >> >> >> -------- Forwarded Message -------- >> Subject: Re: [Crash-utility] crash: page excluded: kernel virtual >> address: ffffffff81e3da50 type: "page_offset_base" >> Date: Wed, 22 Nov 2017 09:35:40 -0500 >> From: Dave Anderson <anderson(a)redhat.com&gt; >> Reply-To: Discussion list for crash utility usage, maintenance and >> development <crash-utility(a)redhat.com&gt; >> To: Discussion list for crash utility usage, maintenance and >> development <crash-utility(a)redhat.com&gt; >> >> >> >> ----- Original Message ----- >>> Hi, >>> >>> I am using the latest crash tool & kernel 4.14 compiled from source, >>> and >>> I got the following error message. As I searched, this is fixed in >>> crash >>> 7.2.0, but I still have it here. So, is anyone has a clue? >> >> It's always going to be a crap-shoot with very recent upstream kernels, >> but you haven't given enough information to determine what the issue is. >> > > Sorry, I am to new dump/crash area. > >> If the vmcore was created by "virsh dump", and the kernel has KASLR >> enabled, >> then it's just not supported at this time. Otherwise, perhaps the >> output of >> "crash -d8" may yield some clues. >> > > I am using kdump to create the vmcore. Actually, I did solve this issue > by add "nokaslr" to kernel parameter when I used the built-in crash of > Fedora. But when turn to the compiled version of my own, that solution > don't work. here is the output of `crash -d8`: >

OK, so you seem to be describing two issues: (1) With KASLR enabled, the vmcore cannot be read by either Fedora's 7.2.0-1 or upstream 7.2.0++. (2) With KASLR disabled, the vmcore can be read by Fedora's 7.2.0-1, but cannot be read by 7.2.0++,

When you run crash on the live system, what happens when: (1) KASLR is enabled, running Fedora's 7.2.0-1 (2) KASLR is enabled, running upstream 7.2.0++ (2) KASLR is disabled, running Fedora's 7.2.0-1 (3) KASLR is disabled, running upstream 7.2.0++ I don't see anything unusual in the output above, presuming the the phys_base of 0 is correct. When you run crash on the live system, what is the output of "help -m | grep phys_base">

I need to compare the crash -d8 output above with the same crash -d8 output from the working session using the same KASLR disabled vmcore, showing at least up until the point that it does the readmem of "page_offset_base" above? Also, if you can make the 2 vmcores available, please send me the details of how to download them in a private email to anderson(a)redhat.com or anderson(a)prospeed.net. If you have modified/rebuilt the Fedora kernel, I would also need a copy of the vmlinux file as well.

> Hi Dave, >   Because I didn't subscribe the list, so please CC me when reply:) My > colleague help to forward your reply to me, so I decide to reply > directly in order to keep this mail in the thread other than new a thread. > > On 11/23/2017 02:04 PM, Fei, Jie/费 杰 wrote: >> >> >> >> -------- Forwarded Message -------- >> Subject: Re: [Crash-utility] crash: page excluded: kernel virtual >> address: ffffffff81e3da50 type: "page_offset_base" >> Date: Wed, 22 Nov 2017 09:35:40 -0500 >> From: Dave Anderson <anderson(a)redhat.com&gt; >> Reply-To: Discussion list for crash utility usage, maintenance and >> development <crash-utility(a)redhat.com&gt; >> To: Discussion list for crash utility usage, maintenance and >> development <crash-utility(a)redhat.com&gt; >> >> >> >> ----- Original Message ----- >>> Hi, >>> >>> I am using the latest crash tool & kernel 4.14 compiled from source, >>> and >>> I got the following error message. As I searched, this is fixed in >>> crash >>> 7.2.0, but I still have it here. So, is anyone has a clue? >> >> It's always going to be a crap-shoot with very recent upstream kernels, >> but you haven't given enough information to determine what the issue is. >> > > Sorry, I am to new dump/crash area. > >> If the vmcore was created by "virsh dump", and the kernel has KASLR >> enabled, >> then it's just not supported at this time. Otherwise, perhaps the >> output of >> "crash -d8" may yield some clues. >> > > I am using kdump to create the vmcore. Actually, I did solve this issue > by add "nokaslr" to kernel parameter when I used the built-in crash of > Fedora. But when turn to the compiled version of my own, that solution > don't work. here is the output of `crash -d8`: > > $ sudo ./crash -d8 /var/crash/127.0.0.1-2017-11-22-18\:21\:42/vmcore > ../linux/vmlinux > > crash 7.2.0++ > Copyright (C) 2002-2017 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > Copyright (C) 1999-2006 Hewlett-Packard Co > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > Copyright (C) 2005, 2011 NEC Corporation > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public License, > and you are welcome to change it and/or distribute copies of it under > certain conditions. Enter "help copying" to see the conditions. > This program has absolutely no warranty. Enter "help warranty" for > details. > > compressed kdump: header->utsname.machine: x86_64 > compressed kdump: memory bitmap offset: 2000 > diskdump_data: > filename: /var/crash/127.0.0.1-2017-11-22-18:21:42/vmcore > flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED) > dfd: 3 > ofp: 0 > machine_type: 62 (EM_X86_64) > > header: 1013930 > signature: "KDUMP " > header_version: 6 > utsname: > sysname: Linux > nodename: IAAS1 > release: 4.14.0 > version: #1 SMP Wed Nov 15 10:32:46 CST 2017 > machine: x86_64 > domainname: (none) > timestamp: > tv_sec: 5a154fac > tv_usec: 0 > status: 2 (DUMP_DH_COMPRESSED_LZO) > block_size: 4096 > sub_hdr_size: 1 > bitmap_blocks: 80 > max_mapnr: 1310208 > total_ram_blocks: 0 > device_blocks: 0 > written_blocks: 0 > current_cpu: 0 > nr_cpus: 4 > tasks[nr_cpus]: 0 > 0 > 0 > 0 > > sub_header: 0 (n/a) > > sub_header_kdump: 1014940 > phys_base: 0 > dump_level: 31 (0x1f) > (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE) > split: 0 > start_pfn: (unused) > end_pfn: (unused) > offset_vmcoreinfo: 5648 (0x1610) > size_vmcoreinfo: 1883 (0x75b) > OSRELEASE=4.14.0 > PAGESIZE=4096 > SYMBOL(init_uts_ns)=ffffffff81e10280 > SYMBOL(node_online_map)=ffffffff82030e80 > SYMBOL(swapper_pg_dir)=ffffffff81e09000 > SYMBOL(_stext)=ffffffff81000000 > SYMBOL(vmap_area_list)=ffffffff81efc470 > SYMBOL(mem_section)=ffffffff82401dc0 > LENGTH(mem_section)=2048 > SIZE(mem_section)=16 > OFFSET(mem_section.section_mem_map)=0 > SIZE(page)=64 > SIZE(pglist_data)=172864 > SIZE(zone)=1664 > SIZE(free_area)=104 > SIZE(list_head)=16 > SIZE(nodemask_t)=128 > OFFSET(page.flags)=0 > OFFSET(page._refcount)=28 > OFFSET(page.mapping)=8 > OFFSET(page.lru)=32 > OFFSET(page._mapcount)=24 > OFFSET(page.private)=48 > OFFSET(page.compound_dtor)=40 > OFFSET(page.compound_order)=44 > OFFSET(page.compound_head)=32 > OFFSET(pglist_data.node_zones)=0 > OFFSET(pglist_data.nr_zones)=172192 > OFFSET(pglist_data.node_start_pfn)=172200 > OFFSET(pglist_data.node_spanned_pages)=172216 > OFFSET(pglist_data.node_id)=172224 > OFFSET(zone.free_area)=192 > OFFSET(zone.vm_stat)=1472 > OFFSET(zone.spanned_pages)=112 > OFFSET(free_area.free_list)=0 > OFFSET(list_head.next)=0 > OFFSET(list_head.prev)=8 > OFFSET(vmap_area.va_start)=0 > OFFSET(vmap_area.list)=48 > LENGTH(zone.free_area)=11 > SYMBOL(log_buf)=ffffffff81e58480 > SYMBOL(log_buf_len)=ffffffff81e5847c > SYMBOL(log_first_idx)=ffffffff823340f8 > SYMBOL(clear_idx)=ffffffff823340cc > SYMBOL(log_next_idx)=ffffffff823340e8 > SIZE(printk_log)=16 > OFFSET(printk_log.ts_nsec)=0 > OFFSET(printk_log.len)=8 > OFFSET(printk_log.text_len)=10 > OFFSET(printk_log.dict_len)=12 > LENGTH(free_area.free_list)=6 > NUMBER(NR_FREE_PAGES)=0 > NUMBER(PG_lru)=5 > NUMBER(PG_private)=12 > NUMBER(PG_swapcache)=9 > NUMBER(PG_slab)=8 > NUMBER(PG_hwpoison)=22 > NUMBER(PG_head_mask)=32768 > NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128 > NUMBER(HUGETLB_PAGE_DTOR)=2 > NUMBER(phys_base)=0 > SYMBOL(init_top_pgt)=ffffffff81e09000 > SYMBOL(node_data)=ffffffff8202c6c0 > LENGTH(node_data)=1024 > KERNELOFFSET=0 > NUMBER(KERNEL_IMAGE_SIZE)=1073741824 > CRASHTIME=1511346092 > offset_note: 4200 (0x1068) > size_note: 3332 (0xd04) > notes_buf: 1015950 > num_prstatus_notes: 4 > notes[0]: 1015950 (NT_PRSTATUS) > si.signo: 0 si.code: 0 si.errno: 0 > cursig: 0 sigpend: 0 sighold: 0 > pid: 114 ppid: 0 pgrp: 0 sid:0 > utime: 0.000000 stime: 0.000000 > cutime: 0.000000 cstime: 0.000000 > ORIG_RAX: ffffffffffffffff fpvalid: 0 > R15: ffffffff82445b80 R14: ffffffff822f3422 > R13: 0000000000000020 R12: 00000000000026f5 > RBP: ffffc90000adbce0 RBX: ffffffff82445b80 > R11: ffffffff822f342d R10: 0000000000000000 > R9: 000000000000000f R8: 0000000000000000 > RAX: 000000799065724b RCX: 0000007990656c1b > RDX: 0000000000000000 RSI: 0000000000000000 > RDI: 0000000000000cdd RIP: ffffffff8188600f > RFLAGS: 0000000000000097 RSP: ffffc90000adbce0 > FS_BASE: 0000000000000000 > GS_BASE: 0000000000000000 > CS: 0010 SS: 0018 DS: 0000 > ES: 0000 FS: 0000 GS: 0000 > notes[1]: 1015ab4 (NT_PRSTATUS) > si.signo: 0 si.code: 0 si.errno: 0 > cursig: 0 sigpend: 0 sighold: 0 > pid: 2030 ppid: 0 pgrp: 0 sid:0 > utime: 0.000000 stime: 0.000000 > cutime: 0.000000 cstime: 0.000000 > ORIG_RAX: ffffffffffffffff fpvalid: 0 > R15: ffff880131999500 R14: ffffffff81f76fa0 > R13: 0000000000000000 R12: 0000000000000007 > RBP: ffffc90001dabdd8 RBX: 0000000000000063 > R11: ffffffff822f342d R10: 0000000000000001 > R9: 0000000000000007 R8: 00000000000002d9 > RAX: 000000000000000f RCX: 0000000000000000 > RDX: 0000000000000000 RSI: ffff88013fa8e138 > RDI: 0000000000000063 RIP: ffffffff81532ad6 > RFLAGS: 0000000000010282 RSP: ffffc90001dabdd8 > FS_BASE: 00007fe58251fb80 > GS_BASE: 0000000000000000 > CS: 0010 SS: 0018 DS: 0000 > ES: 0000 FS: 0000 GS: 0000 > notes[2]: 1015c18 (NT_PRSTATUS) > si.signo: 0 si.code: 0 si.errno: 0 > cursig: 0 sigpend: 0 sighold: 0 > pid: 0 ppid: 0 pgrp: 0 sid:0 > utime: 0.000000 stime: 0.000000 > cutime: 0.000000 cstime: 0.000000 > ORIG_RAX: ffffffffffffffff fpvalid: 0 > R15: ffffffff81f65618 R14: 0000000000000020 > R13: 0000000000000004 R12: 0000000000000003 > RBP: ffffc900006a3e60 RBX: 0000000000000008 > R11: 000000000000298b R10: ffffc900006a3e40 > R9: 0000000000000018 R8: 00000000000054a2 > RAX: 0000000000000020 RCX: 0000000000000001 > RDX: 0000000000000000 RSI: ffffffff81f65480 > RDI: 0000000000000002 RIP: ffffffff81895ddc > RFLAGS: 0000000000000046 RSP: ffffc900006a3e40 > FS_BASE: 0000000000000000 > GS_BASE: 0000000000000000 > CS: 0010 SS: 0018 DS: 0000 > ES: 0000 FS: 0000 GS: 0000 > notes[3]: 1015d7c (NT_PRSTATUS) > si.signo: 0 si.code: 0 si.errno: 0 > cursig: 0 sigpend: 0 sighold: 0 > pid: 0 ppid: 0 pgrp: 0 sid:0 > utime: 0.000000 stime: 0.000000 > cutime: 0.000000 cstime: 0.000000 > ORIG_RAX: ffffffffffffffff fpvalid: 0 > R15: ffffffff81f65618 R14: 0000000000000020 > R13: 0000000000000004 R12: 0000000000000003 > RBP: ffffc900006abe60 RBX: 0000000000000008 > R11: 000000000000037e R10: ffffc900006abe40 > R9: 0000000000000008 R8: 00000000ffffffff > RAX: 0000000000000020 RCX: 0000000000000001 > RDX: 0000000000000000 RSI: ffffffff81f65480 > RDI: 0000000000000003 RIP: ffffffff81895ddc > RFLAGS: 0000000000000046 RSP: ffffc900006abe40 > FS_BASE: 0000000000000000 > GS_BASE: 0000000000000000 > CS: 0010 SS: 0018 DS: 0000 > ES: 0000 FS: 0000 GS: 0000 > snapshot_task: 0 > num_qemu_notes: 0 > NOTE offsets: 1068 (NT_PRSTATUS) > 11cc (NT_PRSTATUS) > 1330 (NT_PRSTATUS) > 1494 (NT_PRSTATUS) > offset_eraseinfo: 0 (0x0) > size_eraseinfo: 0 (0x0) > start_pfn_64: (unused) > end_pfn_64: (unused) > max_mapnr_64: 1310208 (0x13fe00) > > data_offset: 52000 > block_size: 4096 > block_shift: 12 > bitmap: 7f8ff466c010 > bitmap_len: 327680 > max_mapnr: 1310208 (0x13fe00) > dumpable_bitmap: 7f8ff461b010 > byte: 0 > bit: 0 > compressed_page: 10470a0 > curbufptr: 0 > > page_cache_hdr[0]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 1037090 > pg_hit_count: 0 > page_cache_hdr[1]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 1038090 > pg_hit_count: 0 > page_cache_hdr[2]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 1039090 > pg_hit_count: 0 > page_cache_hdr[3]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 103a090 > pg_hit_count: 0 > page_cache_hdr[4]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 103b090 > pg_hit_count: 0 > page_cache_hdr[5]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 103c090 > pg_hit_count: 0 > page_cache_hdr[6]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 103d090 > pg_hit_count: 0 > page_cache_hdr[7]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 103e090 > pg_hit_count: 0 > page_cache_hdr[8]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 103f090 > pg_hit_count: 0 > page_cache_hdr[9]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 1040090 > pg_hit_count: 0 > page_cache_hdr[10]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 1041090 > pg_hit_count: 0 > page_cache_hdr[11]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 1042090 > pg_hit_count: 0 > page_cache_hdr[12]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 1043090 > pg_hit_count: 0 > page_cache_hdr[13]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 1044090 > pg_hit_count: 0 > page_cache_hdr[14]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 1045090 > pg_hit_count: 0 > page_cache_hdr[15]: > pg_flags: 0 () > pg_addr: 0 > pg_bufptr: 1046090 > pg_hit_count: 0 > > page_cache_buf: 1037090 > evict_index: 0 > evictions: 0 > accesses: 0 > cached_reads: 0 > valid_pages: 1036680 > readmem: read_diskdump() > crash: pv_init_ops exists: ARCH_PVOPS > VMCOREINFO: NUMBER(phys_base): 0 -> 0 > gdb ../linux/vmlinux > GNU gdb (GDB) 7.6 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-unknown-linux-gnu"... > GETBUF(288 -> 0) > GETBUF(1500 -> 1) > > FREEBUF(1) > FREEBUF(0) > <readmem: ffffffff81e3da50, KVADDR, "page_offset_base", 8, (FOE), d83568> > <read_diskdump: addr: ffffffff81e3da50 paddr: 1e3da50 cnt: 8> > read_diskdump: PAGE_EXCLUDED: paddr/pfn: 1e3da50/1e3d > crash: page excluded: kernel virtual address: ffffffff81e3da50 type: > "page_offset_base" > > -- > Sincerely, > Cao jin OK, so you seem to be describing two issues: (1) With KASLR enabled, the vmcore cannot be read by either Fedora's 7.2.0-1 or upstream 7.2.0++. (2) With KASLR disabled, the vmcore can be read by Fedora's 7.2.0-1, but cannot be read by 7.2.0++, When you run crash on the live system, what happens when: (1) KASLR is enabled, running Fedora's 7.2.0-1 (2) KASLR is enabled, running upstream 7.2.0++ (2) KASLR is disabled, running Fedora's 7.2.0-1 (3) KASLR is disabled, running upstream 7.2.0++ I don't see anything unusual in the output above, presuming the the phys_base of 0 is correct. When you run crash on the live system, what is the output of "help -m | grep phys_base"? I need to compare the crash -d8 output above with the same crash -d8 output from the working session using the same KASLR disabled vmcore, showing at least up until the point that it does the readmem of "page_offset_base" above? Also, if you can make the 2 vmcores available, please send me the details of how to download them in a private email to anderson(a)redhat.com or anderson(a)prospeed.net. If you have modified/rebuilt the Fedora kernel, I would also need a copy of the vmlinux file as well. We are on Thanksgiving holiday until next Monday, so I won't be able to spend any time looking at this issue until then. Dave .