Re: [Crash-utility] crash: page excluded: kernel virtual address: ffffffff81e3da50 type: "page_offset_base"

Monday, 27 November 2017

Hi Dave,

Information update: I think I found the root cause.

During the test in last week, I always use makedumpfile, use
/etc/kdump.conf as-is:

    core_collector makedumpfile -l --message-level 1 -d 31

And now I am suggested to dump without makedumpfile by using:

    core_collector cp

I get big vmcore file of 3.5G, but crash 7.2.0++ can deal with it
without any error, whether with or without "nokaslr" kernel parameter.

So it seems makedumpfile has problem, and if it does is, I am curious
about that there is no other people report this kind of issue, because
it seems easy to reproduce.

-- 
Sincerely,
Cao jin

On 11/23/2017 10:03 PM, Dave Anderson wrote:
...
> Hi Dave,
> Â  Because I didn't subscribe the list, so please CC me when reply:) My
> colleague help to forward your reply to me, so I decide to reply
> directly in order to keep this mail in the thread other than new a thread.
>
> On 11/23/2017 02:04 PM, Fei, Jie/è´¹ æ° wrote:
>>
>>
>>
>> -------- Forwarded Message --------
>> Subject: 	Re: [Crash-utility] crash: page excluded: kernel virtual
>> address: ffffffff81e3da50 type: "page_offset_base"
>> Date: 	Wed, 22 Nov 2017 09:35:40 -0500
>> From: 	Dave Anderson <anderson(a)redhat.com&gt;
>> Reply-To: 	Discussion list for crash utility usage, maintenance and
>> development <crash-utility(a)redhat.com&gt;
>> To: 	Discussion list for crash utility usage, maintenance and
>> development <crash-utility(a)redhat.com&gt;
>>
>>
>>
>> ----- Original Message -----
>>> Hi,
>>>
>>> I am using the latest crash tool & kernel 4.14 compiled from source,
>>> and
>>> I got the following error message. As I searched, this is fixed in
>>> crash
>>> 7.2.0, but I still have it here. So, is anyone has a clue?
>>
>> It's always going to be a crap-shoot with very recent upstream kernels,
>> but you haven't given enough information to determine what the issue is.
>>
>
> Sorry, I am to new dump/crash area.
>
>> If the vmcore was created by "virsh dump", and the kernel has KASLR
>> enabled,
>> then it's just not supported at this time.  Otherwise, perhaps the
>> output of
>> "crash -d8" may yield some clues.
>>
>
> I am using kdump to create the vmcore. Actually, I did solve this issue
> by add "nokaslr" to kernel parameter when I used the built-in crash of
> Fedora. But when turn to the compiled version of my own, that solution
> don't work. here is the output of `crash -d8`:
>
> $ sudo ./crash -d8 /var/crash/127.0.0.1-2017-11-22-18\:21\:42/vmcore
> ../linux/vmlinux
>
> crash 7.2.0++
> Copyright (C) 2002-2017  Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
> Copyright (C) 1999-2006  Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011  NEC Corporation
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions.  Enter "help copying" to see the conditions.
> This program has absolutely no warranty.  Enter "help warranty" for
> details.
>
> compressed kdump: header->utsname.machine: x86_64
> compressed kdump: memory bitmap offset: 2000
> diskdump_data:
>           filename: /var/crash/127.0.0.1-2017-11-22-18:21:42/vmcore
>              flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
>                dfd: 3
>                ofp: 0
>       machine_type: 62 (EM_X86_64)
>
>             header: 1013930
>            signature: "KDUMP   "
>       header_version: 6
>              utsname:
>                sysname: Linux
>               nodename: IAAS1
>                release: 4.14.0
>                version: #1 SMP Wed Nov 15 10:32:46 CST 2017
>                machine: x86_64
>             domainname: (none)
>            timestamp:
>                 tv_sec: 5a154fac
>                tv_usec: 0
>               status: 2 (DUMP_DH_COMPRESSED_LZO)
>           block_size: 4096
>         sub_hdr_size: 1
>        bitmap_blocks: 80
>            max_mapnr: 1310208
>     total_ram_blocks: 0
>        device_blocks: 0
>       written_blocks: 0
>          current_cpu: 0
>              nr_cpus: 4
>       tasks[nr_cpus]: 0
>                       0
>                       0
>                       0
>
>         sub_header: 0 (n/a)
>
>   sub_header_kdump: 1014940
>            phys_base: 0
>           dump_level: 31 (0x1f)
>
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
>                split: 0
>            start_pfn: (unused)
>              end_pfn: (unused)
>    offset_vmcoreinfo: 5648 (0x1610)
>      size_vmcoreinfo: 1883 (0x75b)
>                       OSRELEASE=4.14.0
>                       PAGESIZE=4096
>                       SYMBOL(init_uts_ns)=ffffffff81e10280
>                       SYMBOL(node_online_map)=ffffffff82030e80
>                       SYMBOL(swapper_pg_dir)=ffffffff81e09000
>                       SYMBOL(_stext)=ffffffff81000000
>                       SYMBOL(vmap_area_list)=ffffffff81efc470
>                       SYMBOL(mem_section)=ffffffff82401dc0
>                       LENGTH(mem_section)=2048
>                       SIZE(mem_section)=16
>                       OFFSET(mem_section.section_mem_map)=0
>                       SIZE(page)=64
>                       SIZE(pglist_data)=172864
>                       SIZE(zone)=1664
>                       SIZE(free_area)=104
>                       SIZE(list_head)=16
>                       SIZE(nodemask_t)=128
>                       OFFSET(page.flags)=0
>                       OFFSET(page._refcount)=28
>                       OFFSET(page.mapping)=8
>                       OFFSET(page.lru)=32
>                       OFFSET(page._mapcount)=24
>                       OFFSET(page.private)=48
>                       OFFSET(page.compound_dtor)=40
>                       OFFSET(page.compound_order)=44
>                       OFFSET(page.compound_head)=32
>                       OFFSET(pglist_data.node_zones)=0
>                       OFFSET(pglist_data.nr_zones)=172192
>                       OFFSET(pglist_data.node_start_pfn)=172200
>                       OFFSET(pglist_data.node_spanned_pages)=172216
>                       OFFSET(pglist_data.node_id)=172224
>                       OFFSET(zone.free_area)=192
>                       OFFSET(zone.vm_stat)=1472
>                       OFFSET(zone.spanned_pages)=112
>                       OFFSET(free_area.free_list)=0
>                       OFFSET(list_head.next)=0
>                       OFFSET(list_head.prev)=8
>                       OFFSET(vmap_area.va_start)=0
>                       OFFSET(vmap_area.list)=48
>                       LENGTH(zone.free_area)=11
>                       SYMBOL(log_buf)=ffffffff81e58480
>                       SYMBOL(log_buf_len)=ffffffff81e5847c
>                       SYMBOL(log_first_idx)=ffffffff823340f8
>                       SYMBOL(clear_idx)=ffffffff823340cc
>                       SYMBOL(log_next_idx)=ffffffff823340e8
>                       SIZE(printk_log)=16
>                       OFFSET(printk_log.ts_nsec)=0
>                       OFFSET(printk_log.len)=8
>                       OFFSET(printk_log.text_len)=10
>                       OFFSET(printk_log.dict_len)=12
>                       LENGTH(free_area.free_list)=6
>                       NUMBER(NR_FREE_PAGES)=0
>                       NUMBER(PG_lru)=5
>                       NUMBER(PG_private)=12
>                       NUMBER(PG_swapcache)=9
>                       NUMBER(PG_slab)=8
>                       NUMBER(PG_hwpoison)=22
>                       NUMBER(PG_head_mask)=32768
>                       NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
>                       NUMBER(HUGETLB_PAGE_DTOR)=2
>                       NUMBER(phys_base)=0
>                       SYMBOL(init_top_pgt)=ffffffff81e09000
>                       SYMBOL(node_data)=ffffffff8202c6c0
>                       LENGTH(node_data)=1024
>                       KERNELOFFSET=0
>                       NUMBER(KERNEL_IMAGE_SIZE)=1073741824
>                       CRASHTIME=1511346092
>          offset_note: 4200 (0x1068)
>            size_note: 3332 (0xd04)
>            notes_buf: 1015950
>   num_prstatus_notes: 4
>             notes[0]: 1015950 (NT_PRSTATUS)
>                       si.signo: 0  si.code: 0  si.errno: 0
>                       cursig: 0  sigpend: 0  sighold: 0
>                       pid: 114  ppid: 0  pgrp: 0  sid:0
>                       utime: 0.000000  stime: 0.000000
>                       cutime: 0.000000  cstime: 0.000000
>                       ORIG_RAX: ffffffffffffffff  fpvalid: 0
>                            R15: ffffffff82445b80  R14: ffffffff822f3422
>                            R13: 0000000000000020  R12: 00000000000026f5
>                            RBP: ffffc90000adbce0  RBX: ffffffff82445b80
>                            R11: ffffffff822f342d  R10: 0000000000000000
>                             R9: 000000000000000f   R8: 0000000000000000
>                            RAX: 000000799065724b  RCX: 0000007990656c1b
>                            RDX: 0000000000000000  RSI: 0000000000000000
>                            RDI: 0000000000000cdd  RIP: ffffffff8188600f
>                         RFLAGS: 0000000000000097  RSP: ffffc90000adbce0
>                        FS_BASE: 0000000000000000
>                        GS_BASE: 0000000000000000
>                             CS: 0010  SS: 0018  DS: 0000
>                             ES: 0000  FS: 0000  GS: 0000
>             notes[1]: 1015ab4 (NT_PRSTATUS)
>                       si.signo: 0  si.code: 0  si.errno: 0
>                       cursig: 0  sigpend: 0  sighold: 0
>                       pid: 2030  ppid: 0  pgrp: 0  sid:0
>                       utime: 0.000000  stime: 0.000000
>                       cutime: 0.000000  cstime: 0.000000
>                       ORIG_RAX: ffffffffffffffff  fpvalid: 0
>                            R15: ffff880131999500  R14: ffffffff81f76fa0
>                            R13: 0000000000000000  R12: 0000000000000007
>                            RBP: ffffc90001dabdd8  RBX: 0000000000000063
>                            R11: ffffffff822f342d  R10: 0000000000000001
>                             R9: 0000000000000007   R8: 00000000000002d9
>                            RAX: 000000000000000f  RCX: 0000000000000000
>                            RDX: 0000000000000000  RSI: ffff88013fa8e138
>                            RDI: 0000000000000063  RIP: ffffffff81532ad6
>                         RFLAGS: 0000000000010282  RSP: ffffc90001dabdd8
>                        FS_BASE: 00007fe58251fb80
>                        GS_BASE: 0000000000000000
>                             CS: 0010  SS: 0018  DS: 0000
>                             ES: 0000  FS: 0000  GS: 0000
>             notes[2]: 1015c18 (NT_PRSTATUS)
>                       si.signo: 0  si.code: 0  si.errno: 0
>                       cursig: 0  sigpend: 0  sighold: 0
>                       pid: 0  ppid: 0  pgrp: 0  sid:0
>                       utime: 0.000000  stime: 0.000000
>                       cutime: 0.000000  cstime: 0.000000
>                       ORIG_RAX: ffffffffffffffff  fpvalid: 0
>                            R15: ffffffff81f65618  R14: 0000000000000020
>                            R13: 0000000000000004  R12: 0000000000000003
>                            RBP: ffffc900006a3e60  RBX: 0000000000000008
>                            R11: 000000000000298b  R10: ffffc900006a3e40
>                             R9: 0000000000000018   R8: 00000000000054a2
>                            RAX: 0000000000000020  RCX: 0000000000000001
>                            RDX: 0000000000000000  RSI: ffffffff81f65480
>                            RDI: 0000000000000002  RIP: ffffffff81895ddc
>                         RFLAGS: 0000000000000046  RSP: ffffc900006a3e40
>                        FS_BASE: 0000000000000000
>                        GS_BASE: 0000000000000000
>                             CS: 0010  SS: 0018  DS: 0000
>                             ES: 0000  FS: 0000  GS: 0000
>             notes[3]: 1015d7c (NT_PRSTATUS)
>                       si.signo: 0  si.code: 0  si.errno: 0
>                       cursig: 0  sigpend: 0  sighold: 0
>                       pid: 0  ppid: 0  pgrp: 0  sid:0
>                       utime: 0.000000  stime: 0.000000
>                       cutime: 0.000000  cstime: 0.000000
>                       ORIG_RAX: ffffffffffffffff  fpvalid: 0
>                            R15: ffffffff81f65618  R14: 0000000000000020
>                            R13: 0000000000000004  R12: 0000000000000003
>                            RBP: ffffc900006abe60  RBX: 0000000000000008
>                            R11: 000000000000037e  R10: ffffc900006abe40
>                             R9: 0000000000000008   R8: 00000000ffffffff
>                            RAX: 0000000000000020  RCX: 0000000000000001
>                            RDX: 0000000000000000  RSI: ffffffff81f65480
>                            RDI: 0000000000000003  RIP: ffffffff81895ddc
>                         RFLAGS: 0000000000000046  RSP: ffffc900006abe40
>                        FS_BASE: 0000000000000000
>                        GS_BASE: 0000000000000000
>                             CS: 0010  SS: 0018  DS: 0000
>                             ES: 0000  FS: 0000  GS: 0000
>        snapshot_task: 0
>       num_qemu_notes: 0
>         NOTE offsets: 1068 (NT_PRSTATUS)
>                       11cc (NT_PRSTATUS)
>                       1330 (NT_PRSTATUS)
>                       1494 (NT_PRSTATUS)
>     offset_eraseinfo: 0 (0x0)
>       size_eraseinfo: 0 (0x0)
>         start_pfn_64: (unused)
>           end_pfn_64: (unused)
>         max_mapnr_64: 1310208 (0x13fe00)
>
>        data_offset: 52000
>         block_size: 4096
>        block_shift: 12
>             bitmap: 7f8ff466c010
>         bitmap_len: 327680
>          max_mapnr: 1310208 (0x13fe00)
>    dumpable_bitmap: 7f8ff461b010
>               byte: 0
>                bit: 0
>    compressed_page: 10470a0
>          curbufptr: 0
>
>  page_cache_hdr[0]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 1037090
>         pg_hit_count: 0
>  page_cache_hdr[1]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 1038090
>         pg_hit_count: 0
>  page_cache_hdr[2]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 1039090
>         pg_hit_count: 0
>  page_cache_hdr[3]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 103a090
>         pg_hit_count: 0
>  page_cache_hdr[4]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 103b090
>         pg_hit_count: 0
>  page_cache_hdr[5]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 103c090
>         pg_hit_count: 0
>  page_cache_hdr[6]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 103d090
>         pg_hit_count: 0
>  page_cache_hdr[7]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 103e090
>         pg_hit_count: 0
>  page_cache_hdr[8]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 103f090
>         pg_hit_count: 0
>  page_cache_hdr[9]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 1040090
>         pg_hit_count: 0
> page_cache_hdr[10]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 1041090
>         pg_hit_count: 0
> page_cache_hdr[11]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 1042090
>         pg_hit_count: 0
> page_cache_hdr[12]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 1043090
>         pg_hit_count: 0
> page_cache_hdr[13]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 1044090
>         pg_hit_count: 0
> page_cache_hdr[14]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 1045090
>         pg_hit_count: 0
> page_cache_hdr[15]:
>             pg_flags: 0 ()
>              pg_addr: 0
>            pg_bufptr: 1046090
>         pg_hit_count: 0
>
>     page_cache_buf: 1037090
>        evict_index: 0
>          evictions: 0
>           accesses: 0
>       cached_reads: 0
>        valid_pages: 1036680
> readmem: read_diskdump()
> crash: pv_init_ops exists: ARCH_PVOPS
> VMCOREINFO: NUMBER(phys_base): 0 -> 0
> gdb ../linux/vmlinux
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> GETBUF(288 -> 0)
>   GETBUF(1500 -> 1)
>
>   FREEBUF(1)
> FREEBUF(0)
> <readmem: ffffffff81e3da50, KVADDR, "page_offset_base", 8, (FOE),
d83568>
> <read_diskdump: addr: ffffffff81e3da50 paddr: 1e3da50 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 1e3da50/1e3d
> crash: page excluded: kernel virtual address: ffffffff81e3da50  type:
> "page_offset_base"
>
> --
> Sincerely,
> Cao jin

 OK, so you seem to be describing two issues:

  (1) With KASLR enabled, the vmcore cannot be read by either
      Fedora's 7.2.0-1 or upstream 7.2.0++.
  (2) With KASLR disabled, the vmcore can be read by Fedora's
      7.2.0-1, but cannot be read by 7.2.0++,

 When you run crash on the live system, what happens when:

  (1) KASLR is enabled, running Fedora's 7.2.0-1
  (2) KASLR is enabled, running upstream 7.2.0++
  (2) KASLR is disabled, running Fedora's 7.2.0-1
  (3) KASLR is disabled, running upstream 7.2.0++

 I don't see anything unusual in the output above, presuming the
 the phys_base of 0 is correct.  When you run crash on the live
 system, what is the output of "help -m | grep phys_base"?

 I need to compare the crash -d8 output above with the same
 crash -d8 output from the working session using the same
 KASLR disabled vmcore, showing at least up until the point that
 it does the readmem of "page_offset_base" above?

 Also, if you can make the 2 vmcores available, please send me
 the details of how to download them in a private email to
 anderson(a)redhat.com or anderson(a)prospeed.net.  If you have
 modified/rebuilt the Fedora kernel, I would also need a copy
 of the vmlinux file as well.

 We are on Thanksgiving holiday until next Monday, so I won't
 be able to spend any time looking at this issue until then.

 Dave

 .

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Re: [Crash-utility] crash: page excluded: kernel virtual address: ffffffff81e3da50 type: "page_offset_base"