[Crash-utility] crash no longer works with x86_64 xen-syms / kdump-vmcore (xen 3.1.2-based)

Hi Oda-san, We have a problem with the RHEL5.2 xen hypervisor kdump vmcores. The RHEL5.2 hypervisor sources have been upgraded to xen version 3.1.2, and the x86_64 hypervisor is now relocatable to a dynamically-determined physical address at boot-time. That being the case, the crash utility cannot possibly work with xen-syms/vmcore pairs without knowing how to translate hypervisor virtual addresses to their physical counterparts. The makedumpfile utility would run into the same problem. I suspect you may have seen this working with upstream xen? This is how I understand it: Prior to RHEL5.2, the hypervisor's text and static data region was direct-mapped. There was (and still is) a direct-mapped region defined like so: DIRECTMAP_VIRT_START 0xffff830000000000 DIRECTMAP_VIRT_END 0xffff840000000000 and the hypervisor text and static data was located inside of that direct-mapped region: # nm -Bn xen-syms-2.6.18-53.el5.debug ffff830000100000 A _start ffff830000100000 A _stext ffff830000100000 T start ffff830000100048 t bad_cpu ffff83000010004f t not_multiboot ffff830000100054 t print_err ffff830000100075 t __start ... [ snip ] ... ffff83000020e6e0 b model ffff83000020e700 b cpu_msrs ffff83000020e900 b saved_lvtpc ffff83000020ea00 b reset_value ffff83000020ea40 b reset_value ffff83000020ea60 b reset_value ffff83000020ea80 A _end # Because hypervisor text/static-data it was direct-mapped, the virtual-to-physical address translation was simple -- only requiring that the DIRECTMAP_VIRT_START identifier (0xffff830000000000) be subtracted from the virtual address, leaving the physical address. In the upgraded RHEL5.2 hypervisor, the kernel text and static data is dynamically relocated at boot time, to a location that is based upon the physical memory layout of the host machine. It has its own 1GB mapped region, that exists here: XEN_VIRT_START 0xffff828c80000000 XEN_VIRT_END 0xffff828cc0000000 # nm -Bn xen-syms-2.6.18-89.el5.debug ffff828c80100000 A _start ffff828c80100000 A _stext ffff828c80100000 T start ffff828c80100014 t __high_start ffff828c801000b7 t int_msg ffff828c801000d7 t hex_msg ... [ snip ] ... ffff828c8024e4a0 b cpu_msrs ffff828c8024e8a0 b saved_lvtpc ffff828c8024eaa0 b cpu_type ffff828c8024eac0 b reset_value ffff828c8024eb00 b reset_value ffff828c8024eb20 b reset_value ffff828c8024eb40 A _end # So translating hypervisor virtual addresses to their physical address can no longer be done by simply subtracting a direct-map identifier like before. Note that the hypervisor code's new version of __pa(), for example, ends up doing this: if ( va > DIRECTMAP_VIRT_START ) return va - DIRECTMAP_VIRT_START; return va - XEN_VIRT_START + xen_phys_start; where the value of "xen_phys_start" is the base physical address of the relocated hypervisor text and static data. Again, there is the problem -- when crash is looking at a xen-syms binary and a vmcore, it does not know the value of "xen_phys_start" and therefore cannot translate hypervisor virtual addresses, and therefore is completely useless. It seems to me that the xen kdump mechanism needs to be modified to store the "xen_phys_start" value in the vmcore someplace? Do have any thoughts on how best to address this? Thanks, Dave