Sorry Dave, 

My mistake. Actually, we have to use 64 bit hypervisor...  The file I show to you was created with xen-64 bit and dom0 32 bit.  :( 

Sorry about my mistakes.

When I tried the crash with --p2m_mfn 000bf969 option. 

x86_xen_kdump_p2m_create: p2m_mfn: bf969
00000000: cccccccc cccccccc cccccccc cccccccc
00000010: cccccccc cccccccc cccccccc cccccccc
00000020: cccccccc cccccccc cccccccc cccccccc
00000030: cccccccc cccccccc cccccccc cccccccc

crash32: read error: physical address: cccccccc000  type: "xen kdump p2m mfn list page"


On Mon, Sep 13, 2010 at 4:03 PM, Dave Anderson <anderson@redhat.com> wrote:

----- "Feng LI" <funglee@gmail.com> wrote:

> Hey Dave,
>
> I attached the crash -d1 output with this email...
>
> Do you think whether anything wrong with my vmcore ?

I'm not sure...

The dom0 "p2m_mfn" value required by the crash utility is
contained in the Xen XEN_ELFNOTE_CRASH_INFO note in the
vmcore header.  That note contains this data structure,
as defined in "include/xen/elfcore.h" in the Xen hypervisor
source tree:

 typedef struct {
     unsigned long xen_major_version;
     unsigned long xen_minor_version;
     unsigned long xen_extra_version;
     unsigned long xen_changeset;
     unsigned long xen_compiler;
     unsigned long xen_compile_date;
     unsigned long xen_compile_time;
     unsigned long tainted;
 #ifdef CONFIG_X86
     unsigned long xen_phys_start;
     unsigned long dom0_pfn_to_mfn_frame_list_list;
 #endif
 } crash_xen_info_t;

When a dom0 crashes, it goes through machine_crash_shutdown()
in the hypervisor's "arch/x86/crash.c" file, where it gets a
pointer to the crash_xen_info structure, and then appends the
CONFIG_X86-only xen_phys_start and dom0_pfn_to_mfn_frame_list_list
fields:

 void machine_crash_shutdown(void)
 {
     crash_xen_info_t *info;

     local_irq_disable();

     nmi_shootdown_cpus();

     disable_IO_APIC();

     hvm_disable();

     info = kexec_crash_save_info();
     info->xen_phys_start = xen_phys_start;
     info->dom0_pfn_to_mfn_frame_list_list =
         arch_get_pfn_to_mfn_frame_list_list(dom0);
 }

And then the crash utility reads the dom0_pfn_to_mfn_frame_list_list
value, and stores it in the "p2m_mfn" field that I referenced in the
last email.

Now, looking at your crash -d1 output, here's the XEN_ELFNOTE_CRASH_INFO,
where it should have picked up the bf969 as the p2m_mfn value:

Elf64_Nhdr:
              n_namesz: 4 ("Xen")
              n_descsz: 80
                n_type: 1000001 (XEN_ELFNOTE_CRASH_INFO)
                        00000003 00000000 00000004 00000000
                        d7beb409 00000000 d7bed168 00000000
                        d7bed138 00000000 d7beb3c5 00000000
                        d7beb3e2 00000000 00000000 00000000
                        d7a00000 00000000 000bf969 00000000
... [ snip ] ...

But it read it as a 0, as evidenced by the "p2m_mfn: 0" shown
below:

... [ snip ] ...

This GDB was configured as "i686-pc-linux-gnu"...

x86_xen_kdump_p2m_create: p2m_mfn: 0

... [ snip ] ...

In the meantime, I provisioned a RHEL5 32-bit x86 system with a 32-bit dom0,
and forced a crash.  As expected, it created a 64-bit ELF vmcore, which
does *not* display the "mismatch" warning message like yours does.  But more
importantly, the XEN_ELFNOTE_CRASH_INFO dump on my vmcore looks like this,
where the p2m_mfn is 2c199:

Elf64_Nhdr:
              n_namesz: 4 ("Xen")
              n_descsz: 40
                n_type: 1000001 (XEN_ELFNOTE_CRASH_INFO)
                        00000003 00000001 0018e97f 0018e98a
                        00190120 0018e932 0018e94f 00000001
                        00000000 0002c199

Note that the fields in your vmcore are 64-bit values, while those above
in the RHEL5 are 32-bit values.  That presumably is due to the fact that
you are running a 64-bit hypervisor?  (whereas my RHEL5 the hypervisor is
a 32-bit)  Are you *sure* that you are running a 32-bit hypervisor?

In any case, the p2m_mfn value in the vmcore header can be overridden
on the crash command line.  What happens if you enter:

 # crash vmlinux vmcore --p2m_mfn bf969

I'm guessing that you'll probably bump into yet another oddity, but it's
worth a shot...