----- Original Message ----- Hello Daisuke and Qian, Sorry for the delay -- I am still catching up on my vacation email, and only got to this one today. Thanks for the two vmcores. Anyway, it was surprising to note is that the two dumpfiles can already can be read with no problem by the crash utility -- with no additional patches to support it. The crash utility just thinks that it's a kdump with some unknown "QEMU" ELF notes: $ crash vmlinux-2.6.32-220.el6.x86_64.gz rhel6.2_x86_64 crash 6.0.8 Copyright (C) 2002-2012 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.3.1 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... KERNEL: vmlinux-2.6.32-220.el6.x86_64.gz DUMPFILE: rhel6.2_x86_64 CPUS: 4 DATE: Thu Aug 2 01:11:23 2012 UPTIME: 02:46:20 LOAD AVERAGE: 0.02, 0.02, 0.00 TASKS: 213 NODENAME: localhost.localdomain RELEASE: 2.6.32-220.el6.x86_64 VERSION: #1 SMP Wed Nov 9 08:03:13 EST 2011 MACHINE: x86_64 (2000 Mhz) MEMORY: 128 MB PANIC: "Oops: 0002 [#1] SMP " (check log for details) PID: 1850 COMMAND: "bash" TASK: ffff88001f1aeb40 [THREAD_INFO: ffff88001f752000] CPU: 0 STATE: TASK_RUNNING (PANIC) crash> bt PID: 1850 TASK: ffff88001f1aeb40 CPU: 0 COMMAND: "bash" #0 [ffff88001f7539e0] machine_kexec at ffffffff81031fcb #1 [ffff88001f753a40] crash_kexec at ffffffff810b8f72 #2 [ffff88001f753b10] oops_end at ffffffff814f04b0 #3 [ffff88001f753b40] no_context at ffffffff8104230b #4 [ffff88001f753b90] __bad_area_nosemaphore at ffffffff81042595 #5 [ffff88001f753be0] bad_area at ffffffff810426be #6 [ffff88001f753c10] __do_page_fault at ffffffff81042dc3 #7 [ffff88001f753d30] do_page_fault at ffffffff814f248e #8 [ffff88001f753d60] page_fault at ffffffff814ef845 [exception RIP: sysrq_handle_crash+22] RIP: ffffffff81325476 RSP: ffff88001f753e18 RFLAGS: 00010096 RAX: 0000000000000010 RBX: 0000000000000063 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000063 RBP: ffff88001f753e18 R8: 0000000000000000 R9: ffffffff8163a940 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000 R13: ffffffff81afad40 R14: 0000000000000286 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #9 [ffff88001f753e20] __handle_sysrq at ffffffff81325732 #10 [ffff88001f753e70] write_sysrq_trigger at ffffffff813257ee #11 [ffff88001f753ea0] proc_reg_write at ffffffff811dae8e #12 [ffff88001f753ef0] vfs_write at ffffffff811765d8 #13 [ffff88001f753f30] sys_write at ffffffff81176fe1 #14 [ffff88001f753f80] system_call_fastpath at ffffffff8100b0f2 RIP: 0000003009cd8450 RSP: 00007fffdddada80 RFLAGS: 00010206 RAX: 0000000000000001 RBX: ffffffff8100b0f2 RCX: 0000000000000400 RDX: 0000000000000002 RSI: 00007f923ca64000 RDI: 0000000000000001 RBP: 00007f923ca64000 R8: 000000000000000a R9: 00007f923ca5d700 R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000000002 R13: 0000003009f9b780 R14: 0000000000000002 R15: 0000003009f9b780 ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b crash> help -n vmcore_data: flags: c0 (KDUMP_LOCAL|KDUMP_ELF64) ndfd: 3 ofp: 1e4addf0 header_size: 3688 num_pt_load_segments: 6 pt_load_segment[0]: file_offset: e68 phys_start: 0 phys_end: 20000000 zero_fill: 0 pt_load_segment[1]: file_offset: 20820e68 phys_start: 20000000 phys_end: 20020000 zero_fill: 0 pt_load_segment[2]: file_offset: 20840e68 phys_start: 20020000 phys_end: 20040000 zero_fill: 0 pt_load_segment[3]: file_offset: 20000e68 phys_start: 20040000 phys_end: 20840000 zero_fill: 0 pt_load_segment[4]: file_offset: 20810e68 phys_start: 20840000 phys_end: 20850000 zero_fill: 0 pt_load_segment[5]: file_offset: 20800e68 phys_start: 20850000 phys_end: 20860000 zero_fill: 0 elf_header: 1c8d6fe0 elf32: 0 notes32: 0 load32: 0 elf64: 1c8d6fe0 notes64: 1c8d7020 load64: 1c8d7058 nt_prstatus: 1c8d71a8 nt_prpsinfo: 0 nt_taskstruct: 0 task_struct: 0 page_size: 0 switch_stack: 0 xen_kdump_data: (unused) num_prstatus_notes: 4 vmcoreinfo: 0 size_vmcoreinfo: 0 nt_prstatus_percpu: 000000001c8d71a8 000000001c8d730c 000000001c8d7470 000000001c8d75d4 Elf64_Ehdr: e_ident: \177ELF e_ident[EI_CLASS]: 2 (ELFCLASS64) e_ident[EI_DATA]: 1 (ELFDATA2LSB) e_ident[EI_VERSION]: 1 (EV_CURRENT) e_ident[EI_OSABI]: 0 (ELFOSABI_SYSV) e_ident[EI_ABIVERSION]: 0 e_type: 4 (ET_CORE) e_machine: 62 (EM_X86_64) e_version: 1 (EV_CURRENT) e_entry: 0 e_phoff: 40 e_shoff: 0 e_flags: 0 e_ehsize: 40 e_phentsize: 38 e_phnum: 7 e_shentsize: 0 e_shnum: 0 e_shstrndx: 0 Elf64_Phdr: p_type: 4 (PT_NOTE) p_offset: 456 (1c8) p_vaddr: 0 p_paddr: 0 p_filesz: 3232 (ca0) p_memsz: 3232 (ca0) p_flags: 0 () p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 3688 (e68) p_vaddr: 0 p_paddr: 0 p_filesz: 536870912 (20000000) p_memsz: 536870912 (20000000) p_flags: 0 () p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 545394280 (20820e68) p_vaddr: 0 p_paddr: 20000000 p_filesz: 131072 (20000) p_memsz: 131072 (20000) p_flags: 0 () p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 545525352 (20840e68) p_vaddr: 0 p_paddr: 20020000 p_filesz: 131072 (20000) p_memsz: 131072 (20000) p_flags: 0 () p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 536874600 (20000e68) p_vaddr: 0 p_paddr: 20040000 p_filesz: 8388608 (800000) p_memsz: 8388608 (800000) p_flags: 0 () p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 545328744 (20810e68) p_vaddr: 0 p_paddr: 20840000 p_filesz: 65536 (10000) p_memsz: 65536 (10000) p_flags: 0 () p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 545263208 (20800e68) p_vaddr: 0 p_paddr: 20850000 p_filesz: 65536 (10000) p_memsz: 65536 (10000) p_flags: 0 () p_align: 0 Elf64_Nhdr: n_namesz: 5 ("CORE") n_descsz: 336 n_type: 1 (NT_PRSTATUS) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000001 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 000000000309f000 ffffffffffffffff 0000000000000000 ffffffff81c00480 ffffffff81a01ec8 0000000000000000 0000000000000000 00000013911d5f29 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000001 ffffffff81dd5228 0000000000000000 ffffffff810375ab 0000000000000010 0000000000000246 ffffffff81a01ec8 0000000000000018 0000000000000000 ffff880003200000 0000000000000018 0000000000000018 0000000000000000 0000000000000000 0000000000000000 Elf64_Nhdr: n_namesz: 5 ("CORE") n_descsz: 336 n_type: 1 (NT_PRSTATUS) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000002 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffffffff81bfed40 0000000000000000 00000000fffffffe ffff880002287ef8 ffff880002287e88 000000000000000c ffff88001e6abe78 000009149661fc2b ffff880002287e54 ffff880002287e50 ffffffff81a93760 0000000080802001 0000000000000000 00000000000000ff 00000000000000f0 0000000000000000 ffffffff810375ba 0000000000000010 0000000000000002 ffff880002287e88 0000000000000018 0000000000000000 ffff880002280000 0000000000000018 0000000000000018 0000000000000000 0000000000000000 0000000000000000 Elf64_Nhdr: n_namesz: 5 ("CORE") n_descsz: 336 n_type: 1 (NT_PRSTATUS) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000003 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffffffff81bfed40 0000000000000000 00000000fffffffe ffff880002307ef8 ffff880002307e88 000000000000000c ffff88001e6dfe78 000009143aed494c ffff880002307e54 ffff880002307e50 ffffffff81a93760 0000000080802001 0000000000000000 00000000000000ff 00000000000000f0 0000000000000000 ffffffff810375ba 0000000000000010 0000000000000002 ffff880002307e88 0000000000000018 0000000000000000 ffff880002300000 0000000000000018 0000000000000018 0000000000000000 0000000000000000 0000000000000000 Elf64_Nhdr: n_namesz: 5 ("CORE") n_descsz: 336 n_type: 1 (NT_PRSTATUS) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000004 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffffffff81bfed40 0000000000000000 00000000fffffffe ffff880002387ef8 ffff880002387e88 000000000000000c 0000000000000000 0000091497285969 ffff880002387e54 ffff880002387e50 ffffffff81a93760 0000000080802001 0000000000000000 00000000000000ff 00000000000000f0 0000000000000000 ffffffff810375ba 0000000000000010 0000000000000046 ffff880002387e88 0000000000000018 0000000000000000 ffff880002380000 0000000000000018 0000000000000018 0000000000000000 0000000000000000 0000000000000000 Elf64_Nhdr: n_namesz: 5 ("QEMU") n_descsz: 432 n_type: 0 (?) 000001b000000001 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000001 ffffffff81dd5228 ffffffff81a01ec8 ffffffff81a01ec8 0000000000000000 0000000000000000 00000013911d5f29 0000000000000000 ffffffff81c00480 0000000000000000 ffffffffffffffff 000000000309f000 ffffffff810375ab 0000000000000246 ffffffff00000010 0000000000a09b00 0000000000000000 ffffffff00000018 0000000000c09300 0000000000000000 ffffffff00000018 0000000000c09300 0000000000000000 ffffffff00000000 0000000000000000 0000000000000000 ffffffff00000000 0000000000000000 ffff880003200000 ffffffff00000018 0000000000c09300 0000000000000000 ffffffff00000000 0000000000000000 0000000000000000 0000208700000040 0000000000008b00 ffff880003213b40 0000007f00000000 0000000000000000 ffff880003204000 00000fff00000000 0000000000000000 ffffffff81dd2000 000000008005003b 0000000000000000 0000000001b2e000 0000000007b18000 00000000000006f0 Elf64_Nhdr: n_namesz: 5 ("QEMU") n_descsz: 432 n_type: 0 (?) 000001b000000001 ffffffff81a93760 000000000000000c 0000000080802001 0000000000000000 00000000000000ff 00000000000000f0 ffff880002287e88 ffff880002287e88 ffff880002287e50 ffff880002287e54 000009149661fc2b ffff88001e6abe78 ffff880002287ef8 00000000fffffffe 0000000000000000 ffffffff81bfed40 ffffffff810375ba 0000000000000002 ffffffff00000010 0000000000a09b00 0000000000000000 ffffffff00000018 0000000000c09300 0000000000000000 ffffffff00000018 0000000000c09300 0000000000000000 ffffffff00000000 0000000000000000 0000000000000000 ffffffff00000000 0000000000000000 ffff880002280000 ffffffff00000018 0000000000c09300 0000000000000000 ffffffff00000000 0000000000000000 0000000000000000 0000208700000040 0000000000008b00 ffff880002293b40 0000007f00000000 0000000000000000 ffff880002284000 00000fff00000000 0000000000000000 ffffffff81dd2000 000000008005003b 0000000000000000 0000000002162570 000000001aab8000 00000000000006e0 Elf64_Nhdr: n_namesz: 5 ("QEMU") n_descsz: 432 n_type: 0 (?) 000001b000000001 ffffffff81a93760 000000000000000c 0000000080802001 0000000000000000 00000000000000ff 00000000000000f0 ffff880002307e88 ffff880002307e88 ffff880002307e50 ffff880002307e54 000009143aed494c ffff88001e6dfe78 ffff880002307ef8 00000000fffffffe 0000000000000000 ffffffff81bfed40 ffffffff810375ba 0000000000000002 ffffffff00000010 0000000000a09b00 0000000000000000 ffffffff00000018 0000000000c09300 0000000000000000 ffffffff00000018 0000000000c09300 0000000000000000 ffffffff00000000 0000000000000000 0000000000000000 ffffffff00000000 0000000000000000 ffff880002300000 ffffffff00000018 0000000000c09300 0000000000000000 ffffffff00000000 0000000000000000 0000000000000000 0000208700000040 0000000000008b00 ffff880002313b40 0000007f00000000 0000000000000000 ffff880002304000 00000fff00000000 0000000000000000 ffffffff81dd2000 000000008005003b 0000000000000000 00007fd1a029c000 000000001d5c7000 00000000000006e0 Elf64_Nhdr: n_namesz: 5 ("QEMU") n_descsz: 432 n_type: 0 (?) 000001b000000001 ffffffff81a93760 000000000000000c 0000000080802001 0000000000000000 00000000000000ff 00000000000000f0 ffff880002387e88 ffff880002387e88 ffff880002387e50 ffff880002387e54 0000091497285969 0000000000000000 ffff880002387ef8 00000000fffffffe 0000000000000000 ffffffff81bfed40 ffffffff810375ba 0000000000000046 ffffffff00000010 0000000000a09b00 0000000000000000 ffffffff00000018 0000000000c09300 0000000000000000 ffffffff00000018 0000000000c09300 0000000000000000 ffffffff00000000 0000000000000000 0000000000000000 ffffffff00000000 0000000000000000 ffff880002380000 ffffffff00000018 0000000000c09300 0000000000000000 ffffffff00000000 0000000000000000 0000000000000000 0000208700000040 0000000000008b00 ffff880002393b40 0000007f00000000 0000000000000000 ffff880002384000 00000fff00000000 0000000000000000 ffffffff81dd2000 000000008005003b 0000000000000000 00007f981fe51000 000000001f214000 00000000000006e0 crash> That being the case, the question that immediately arises is: Why bother to create a new special dumpfile "type"? When I quickly reviewed Qian's original patch, I was not aware that the dumpfile is already a "clone" of a kdump, and that crash already supports it with no additional patches required. Knowing that, I'd much rather prefer to keep things simpler, and avoid creating a new dumpfile type at all. Also, when I suggested creating a new dumpfile type, I forgot how ugly and confusing the repetitive changes such as this are: } else if (is_xendump(argv[optind])) { - if (pc->flags & MEMORY_SOURCES) { + if (pc->flags & MEMORY_SOURCES || pc->flags2 & QEMU_MEM_DUMP) { error(INFO, "too many dumpfile arguments\n"); program_usage(SHORT_FORM); I should have suggested moving one of the currently-existing pc->flags bits into pc->flags2, and keeping all the dumpfile types in pc->flags. Or at least create a new macro to replace all the usages of "(pc->flags & MEMORY_SOURCES)". My mistake, Qian -- sorry about that... But -- it seems that we can just leave the dumpfile type as a "KDUMP_ELF32/KDUMP_ELF64", and then based upon the existence of a "QEMU" note, just set a QEMU_MEM_DUMP pc->flags2 bit that can be used everywhere that you're interested in accessing the "QEMU" notes/registers section? Wouldn't that be far simpler? Dave