Hello, From: Dave Anderson <anderson(a)redhat.com&gt; Subject: Re: [Crash-utility] kdump format may be updated Date: Tue, 17 Oct 2006 09:01:32 -0400 Yes, I remember the discussion and I think it is possible to make headers. Now, Magnus is cleaning the patch. He and I discussed, but I'm not enough to convince him that dom0 information is need for crash. I know you don't want to treat xen binary file with crash, but I'm not clear why. Please discuss with him directly to make up xen kdump file formats. The patch will be merged into xen-3.0.4. I hope we can find solution before merge. -- Kazuo Moriwaka <moriwaka(a)valinux.co.jp&gt;

Hi, I'm afraid it may be deleted while patch-cleaning process.. Yes, I think so. I (and Magnus) are not clean about mismatch factor for getting 'idle_pg_table' and 'dom0->shared_info->pfn_to_mfn_frame_list_list' from xen binary in crash is_netdump routine. I think appealing this terribleness may help keep ELF Core headers. Thanks, -- Kazuo Moriwaka <moriwaka(a)valinux.co.jp&gt;

Magnus Damm wrote: Believe me -- if you could tell me how, I would do it, and we could avoid this discussion... A bit of history -- and I don't mean to "dumb-down" the discussion... If you just give me a dump of memory, I can make the crash utility work with it, as long as I can take a physical address and turn it into a file offset into the dumpfile. The first set of physical memory reads that crash does take the unity-mapped virtual addresses of key kernel data, strip off the identifier and pass the physical address to the read function of the particular memory device, i.e., be it the live memory, netdumps, diskdumps, LKCD dumps, mcore dumps, kdumps, xen dumps, and xen "xm save" files. The issue at hand with the xen kdump vmcore is that its ELF header describes physical memory in machine memory terms. To use that vmcore with respect to dom0 instead of the xen binary, for each required physical address derived from a dom0 kernel virtual address, the physical address is a pseudo-physical address with no correlation with the physical (machine) memory descriptors in the ELF header. So, given that the dom0 pseudo-physical address needs to be translated into a machine address, I need to be able to find my way to the phys_to_machine_mapping array. From that point on, it's becomes a matter of searching the array for the desired pseudo-physical address, getting the associated machine address, and then using the PT_LOAD segments of the ELF header to find the memory. To find the phys_to_machine_mapping array, there are two keys to Pandora's box: (1) the dom0 cr3 value -- which in a writable page table kernel, will contain an mfn value. With that starting point, a page table walk can be initiated for the "phys_to_machine_mapping" virtual address. (2) alternatively, given the dom0 pfn_to_mfn_frame_list_list mfn, I also have a starting point in order to reconstruct the phys_to_machine_mapping array. Either one works. I preferred #2 because it would presumably work for both writable and shadow page table kernels. But, I've never done any work with shadow page table kernels (Red Hat is going with writable...), so I don't know what the ramifications are for those kernels. In fact, my original suggestion was for an ELF note with an array of cr3's or pfn_to_mfn_frame_list_list mfn's, i.e., for dom0 and all the other domUs running on the system. That would be an awesome capability -- a single vmcore that could be used against the xen binary using gdb, or with the crash utility for analyszing dom0 or any of the other domU sessions. But, there apparently was an issue with the idea of having an ELF note with an undetermined size. Anyway, like I mentioned to Kazuo, I'd be happy with just the dom0 "key"... Also, in the ELF NT_PRSTATUS note, the cr3 value is not stored since it consists of a processor-specific user_regs_struct. So if you were to append it there, I could use that instead. Your initial patch seems to have put it both there and in the new NT_XEN_DOM0_CR3 note: +/* The cr3 for dom0 on each of its vcpus + * It is added as ELF_Prstatus prstatus.pr_reg[ELF_NGREG-1)], where + * prstatus is the data of the elf note, and ELF_NGREG was extended + * by one to allow extra space. + * This code runs after all cpus except the crashing one have + * been shutdown so as to avoid having to hold domlist_lock, + * as locking after a crash is playing with fire */ + buf = append_elf_note(buf, "Xen Domanin-0 CR3", + NT_XEN_DOM0_CR3, &cr3, 4); But, again, only for x86. Well, to answer your question with a question: Why should it be required if it could be so easily avoided? As Henry David Thoreau said, "Simplicity, simplicity, simplicity..." My two cents, Dave

Magnus Damm wrote: I don't know. If I look at the xen-syms file, the VirtAddr and PhysAddr in the PT_LOAD segment are both 0xff100000, which doesn't make sense to me: Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align LOAD 0x001000 0xff100000 0xff100000 0x92658 0x92658 RWE 0x1000 But anyway, I would presume that the the crash_notes contents in the xen binary contains registers that only make sense with respect to xen, and even if you put the xen cr3 value there, it doesn't have any relationship to the cr3 value required for translating dom0 linux kernel virtual addresses. In your patch here: [Xen-devel] [PATCH 03/04] Kexec / Kdump: x86_32 specific code the dom0 cr3 is located by the find_dom0_cr3() function, which has to walk the xen binary's "domain_list" +void find_dom0_cr3(void) +{ + struct domain *d; + struct vcpu *v; + uint32_t *buf; + uint32_t cr3; + Elf_Note note; + + /* Don't need to grab domlist_lock as we are the only thing running */ + + /* No need to traverse domain_list, as dom0 is always first */ + d = domain_list; + BUG_ON(d->domain_id); + + for_each_vcpu ( d, v ) { + if ( test_bit(_VCPUF_down, &v->vcpu_flags) ) + continue; + buf = (uint32_t *)per_cpu(crash_notes, v->processor); + if (!buf) /* XXX: Can this ever occur? */ + continue; + + memcpy(&note, buf, sizeof(Elf_Note)); + buf += (sizeof(Elf_Note) +3)/4 + (note.namesz + 3)/4 + + (note.descsz + 3)/4; + + /* XXX: This probably doesn't take into account shadow mode, + * but that might not be a problem */ + cr3 = pagetable_get_pfn(v->arch.guest_table); + + buf = append_elf_note(buf, "Xen Domanin-0 CR3", + NT_XEN_DOM0_CR3, &cr3, 4); + final_note(buf); + + printk("domain:%i vcpu:%u processor:%u cr3:%08x\n", + d->domain_id, v->vcpu_id, v->processor, cr3); + } +} + So when you ask, can I "look up the crash_notes symbol and find the saved registers there?", I'm presuming that you are talking about the crash_notes variable above? But if you're planning on removing the code above, why would there be any crash_notes variable remaining? In any case, you can see that what is needed is the per-domain cr3 value, which is embedded in the xen-binary data. Or, as I mentioned before, alternatively, the pfn_to_mfn_frame_list_list value from the per-domain arch_shared_info also would work: struct arch_shared_info { unsigned long max_pfn; /* max pfn that appears in table */ /* Frame containing list of mfns containing list of mfns containing p2m. */ xen_pfn_t pfn_to_mfn_frame_list_list; unsigned long nmi_reason; uint64_t pad[32]; }; But again, there's no easy way for the crash utility to dig them out of a completely foreign binary's. That's exactly right... Your knowledge is a hell of lot less limited than mine... ;-) Anyway, yes, it appears that the "domain_list" global is the key that leads to either the per-domain pfn_to_mfn_frame_list_list or the per-domain cr3 value. With respect to the per-domain pfn_to_mfn_frame_list_list location, in the xen sources, there's the following "shared_info" structure, that contains the data that's shared between the xen binary and and each domain. And this structure contains the "arch_shared_info" structure I showed above -- which contains the per-domain pfn_to_mfn_frame_list_list value: /* * Xen/kernel shared data -- pointer provided in start_info. * * This structure is defined to be both smaller than a page, and the * only data on the shared page, but may vary in actual size even within * compatible Xen versions; guests should not rely on the size * of this structure remaining constant. */ struct shared_info { struct vcpu_info vcpu_info[MAX_VIRT_CPUS]; /* * A domain can create "event channels" on which it can send and receive * asynchronous event notifications. There are three classes of event that * are delivered by this mechanism: * 1. Bi-directional inter- and intra-domain connections. Domains must * arrange out-of-band to set up a connection (usually by allocating * an unbound 'listener' port and avertising that via a storage service * such as xenstore). * 2. Physical interrupts. A domain with suitable hardware-access * privileges can bind an event-channel port to a physical interrupt * source. * 3. Virtual interrupts ('events'). A domain can bind an event-channel * port to a virtual interrupt source, such as the virtual-timer * device or the emergency console. * * Event channels are addressed by a "port index". Each channel is * associated with two bits of information: * 1. PENDING -- notifies the domain that there is a pending notification * to be processed. This bit is cleared by the guest. * 2. MASK -- if this bit is clear then a 0->1 transition of PENDING * will cause an asynchronous upcall to be scheduled. This bit is only * updated by the guest. It is read-only within Xen. If a channel * becomes pending while the channel is masked then the 'edge' is lost * (i.e., when the channel is unmasked, the guest must manually handle * pending notifications as no upcall will be scheduled by Xen). * * To expedite scanning of pending notifications, any 0->1 pending * transition on an unmasked channel causes a corresponding bit in a * per-vcpu selector word to be set. Each bit in the selector covers a * 'C long' in the PENDING bitfield array. */ unsigned long evtchn_pending[sizeof(unsigned long) * 8]; unsigned long evtchn_mask[sizeof(unsigned long) * 8]; /* * Wallclock time: updated only by control software. Guests should base * their gettimeofday() syscall on this wallclock-base value. */ uint32_t wc_version; /* Version counter: see vcpu_time_info_t. */ uint32_t wc_sec; /* Secs 00:00:00 UTC, Jan 1, 1970. */ uint32_t wc_nsec; /* Nsecs 00:00:00 UTC, Jan 1, 1970. */ struct arch_shared_info arch; }; The best "common-code" example I see in the xen sources accessess the shared_info structure above like so: void evtchn_set_pending(struct vcpu *v, int port) { struct domain *d = v->domain; shared_info_t *s = d->shared_info; So looking at your currently existing find_dom0_cr3() function, the vcpu pointer can be pulled the same way from the domain_list: +void find_dom0_cr3(void) +{ + struct domain *d; + struct vcpu *v; + uint32_t *buf; + uint32_t cr3; + Elf_Note note; + + /* Don't need to grab domlist_lock as we are the only thing running */ + + /* No need to traverse domain_list, as dom0 is always first */ + d = domain_list; + BUG_ON(d->domain_id); + + for_each_vcpu ( d, v ) { That's correct. As I explained above, I don't see how? I can't maneuver around following multiple data structure linkages -- using data structures that crash knows nothing about. (Not to mention knowing how exactly the xen virtual-to-physical translation works...) But I'm still confused about that -- why would the "crash_notes" exist in the xen sources/binary if you're not going to put them in the ELF header of a resultant xen dumpfile? What exactly is going to be put into the dumpfile's ELF header? Dave

Magnus Damm wrote: Actually, given that the crash utility is only interested in the specifics of the dom0 kernel, it has no interest in physical cpus. If you're specifically interested in debugging a crash that occurred while operating in the xen binary, you're going to want to use gdb on the vmcore file with xen-syms-xxx namelist file. You can still run crash on the same vmcore to find out what was going on in the dom0 kernel, but there's no awareness of the xen hypervisor underpinnings; you'll just get the state of the dom0 kernel at the time of the crash. But I would guess-timate that the majority of the crashes are going to have occurred in the dom0 kernel, and not while running in the hypervisor. Now, given that that the crash_notes context contains registers that are indexed by the physical cpu number, well, that's not helpful to crash's needs with respect to dom0. That's why you guys must have created the additional NT_XEN_DOM0_CR3 ELF note. I guess I understand why you feel it's a burden to continue the maintenance of such a thing, but given that the panic can occur either while operating in the dom0 kernel or while in the xen hypervisor code, it makes perfect sense (to me) to make a minimal effort by including an indication of the dom0 cr3 or dom0 pfn_to_mfn_frame_list_list value in the vmcore. Perhaps you consider it a case of the tail wagging the dog, but to me, it would be more a case of accomodating the needs of the consumer... ;-) I don't care about that. All I need is a starting point for translating dom0 kernel virtual addresses. And that is either a dom0 cr3 value or the domain's pfn_to_mfn_frame_list_list value. No! Now we're getting complex -- I'm pretty sure I don't know what you're talking about here... Or how it can possibly lead to a dom0 cr3 or pfn_to_mfn_frame_list_list value? > That's good, isn't it? If I've understood things right it's possible to > locate the data you need using the domain list symbol? Yeah... To clarify, it's possible for *you*, i.e., the kexec/kdump code, to locate the data that way. The crash utility, using the vmlinux/vmcore file pair, doesn't know anything about what the "domain_list" is, the structures that it uses/links-to. And even if it did, it wouldn't know how to find it in the vmcore file. > Yeah, I agree that navigating around those structures seems rather > painful. But OTOH, if you want to know things that only the internals > can tell you, you need to be able to parse them, right? But maybe you > only want to cover the "simple" dom0 case. (Simple yeah right) That's right -- crash is *only* interested in the dom0 case; again it's clueless about the hypervisor, and rightly so. It's just such a unique case. It's like trying to debug "ls" using a "cat" binary, where the core file is usable for debugging either one. Thanks, Dave

Magnus Damm wrote: That's fine -- the crash utility will ignore the registers in the NT_PRSTATUS notes if it can determine that the vmcore is a hypervisor/dom0/kdump-type dumpfile. In my prototype, it does just that when it sees the unique NT_XEN_DOM0_CR3 note. (crash doesn't really need the NR_PRSTATUS register contents, because it can find them elsewhere if need be.) That's exactly right -- I requested dom0's cr3 or pfn_to_mfn_list_list value, and Moriwaka-san provided me with a prototype vmcore that contained the NT_XEN_DOM0_CR3 ELF note. I presume that the i386 vmcore he sent me was also usable with gdb and xen-syms. Here's what the header looks like: $ readelf -a vmcore ELF Header: Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 Class: ELF32 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: CORE (Core file) Machine: Intel 80386 Version: 0x1 Entry point address: 0x0 Start of program headers: 52 (bytes into file) Start of section headers: 0 (bytes into file) Flags: 0x0 Size of this header: 52 (bytes) Size of program headers: 32 (bytes) Number of program headers: 5 Size of section headers: 0 (bytes) Number of section headers: 0 Section header string table index: 0 There are no sections in this file. Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align NOTE 0x0000d4 0x00000000 0x00000000 0x00190 0x00190 0 LOAD 0x000264 0xc0000000 0x00000000 0xa0000 0xa0000 RWE 0 LOAD 0x0a0264 0xc0100000 0x00100000 0x3f00000 0x3f00000 RWE 0 LOAD 0x3fa0264 0xc8000000 0x08000000 0x30000000 0x30000000 RWE 0 LOAD 0x33fa0264 0xffffffff 0x38000000 0x7ffb000 0x7ffb000 RWE 0 There is no dynamic segment in this file. There are no relocations in this file. There are no unwind sections in this file. No version information found in this file. Notes at offset 0x000000d4 with length 0x00000190: Owner Data size Description CORE 0x00000090 NT_PRSTATUS (prstatus structure) Xen Domanin-0 CR3 0x00000004 Unknown note type: (0x10000001) CORE 0x00000090 NT_PRSTATUS (prstatus structure) Xen Domanin-0 CR3 0x00000004 Unknown note type: (0x10000001) $ Definitely not. In the interest of simplicity, the idea is to keep things the way they have always been, i.e., "crash vmlinux vmcore", and having to drag out the relevant xen-syms binary defeats that concept. And that's not to mention having to read it, figure out how to translate hypervisor virtual addresses, then navigate around the vmcore to find the location of a symbol, having to deal with the following of data structures that may change over time -- to eventually find what's needed... But I'm not sure what good would that do? The "domain_list" symbol is only the beginning of the structure chain... Looks like a reasonable spot to stuff cr3 (or pfn_to_mfn_frame_list_list)? (i.e. stuffing the dom0 cr3 at the end of the dom0 NT_PRSTATUS register dump(s), like your current patch talks about) With respect to the PT_LOAD segments, the crash utility pretty much ignores the p_vaddr values -- it's particularly interested in the p_paddr values. And once able to translate a unity-mapped kernel virtual address into a physical address, it can find that physical address by checking the p_paddr/p_filesz values in each PT_LOAD segment. (Again -- that only works with xen kernels if the pseudo physical address can be translated into a machine address, and then a vmcore file location -- hence the pre-requisite requirement for either the dom0 cr3 or pfn_to_mfn_list_list value.) The one exception to "ignore-the-p_vaddr" rule is for the new relocatable x86_64 kernels, where the PT_LOAD mapping for the kernel's __START_KERNEL_map region needs to be looked at in order to figure out where the kernel was physically loaded, because in relocatable kernels, unity-mapping can no longer be done by simply stripping off the PAGE_OFFSET value. Relocatable kernels are being introduced so that there is no need for separate "first-kernel" and the secondary kexec'd "kdump-kernel". But AFAIK, the new kernel relocation scheme shouldn't infect xen x86_64 kernels, so I don't believe that would ever be an issue. In other words, xen kernels should remain unity-mapped in the traditional manner, albeit with pseudo-physical addresses instead of machine addresses. Again, I *believe* that to be true, since there's no reason to relocate them. Or this would be good... ;-) Nothing sounds bad to me! What would be really helpful is if, during your development, you could provide me with sample vmlinux/vmcore pairs that I could work with? Just so I don't have to scramble at the end of it all, and only then find out that there's a "gotcha" that we're not considering. Thanks, Dave

Magnus Damm wrote: [ agree-to-agree love-fest... ] > I will contact you when I've updated and tested my code. It will take a > while but I guess within two weeks. > One final question: Is it ok with you that we only use 64-bit ELF file > format for vmcore under Xen? That's definitely fine with me -- in fact, I much prefer that so that we can seamlessly handle 4G+ 32-bit dom0's. With respect to the Red Hat kdump facility, I thought we had decided to unconditionally enforce the 64-bit ELF format, but I see that in the current kexec-ktools package that the /etc/sysconfig/kdump file still contains this: ... # Any additional kexec arguments required. In most situations, this should # be left empty # # Example: # KEXEC_ARGS="--elf32-core-headers" KEXEC_ARGS=" --args-linux" ... And the kexec-tools code still has support for --elf32-core-headers. But anyway, I'm not the arbiter of that issue. I'll just make crash work with anything that comes down the pike. So, it sounds like we have a plan -- thanks for listening! Dave