Worth, Kevin wrote: To be honest with you I'm surprised it comes up at all... If you do a "crash -d7 vmlinux vmcore", amongst the reams of debug data you will see the readmem() that failed just prior to the "WARNING: cannot access vmalloc'd module memory". And that will probably be the very first access of a vmalloc'd virtual memory address. Probably it's best to enter "crash -d7 vmlinux vmcore > /tmp/junk", and then enter "q" to silently kill the session. For that matter, once you come up, I'm guessing that user virtual address translation will fail as well. Come up as you did above, do a "vm" command on the "bash" task, and then a "vtop" on a user virtual address. Like this example: crash> vm PID: 25479 TASK: f6f2aaa0 CPU: 3 COMMAND: "bash" MM PGD RSS TOTAL_VM f6e3d740 f745c980 1560k 4608k VMA START END FLAGS FILE f6c115f4 110000 112000 75 /lib/ f7212f94 112000 113000 100071 /lib/ f78cd0cc 113000 114000 100073 /lib/ f6954d84 584000 585000 8000075 f7241bcc 5b1000 5b4000 75 /lib/libtermcap.so.2.0.8.#prelink#.YYRDOu f7212a14 5b4000 5b5000 100073 /lib/libtermcap.so.2.0.8.#prelink#.YYRDOu f6e1a64c 61a000 623000 75 /lib/libnss_files-2.5.so f73f738c 623000 624000 100071 /lib/libnss_files-2.5.so f6eb79bc 624000 625000 100073 /lib/libnss_files-2.5.so f7212f3c 719000 733000 875 /lib/ f721238c 733000 734000 100871 /lib/ f72e2b1c 734000 735000 100873 /lib/ f73f7964 ab5000 bf2000 75 /lib/ f6c11ee4 bf2000 bf4000 100071 /lib/ f73f7ee4 bf4000 bf5000 100073 /lib/ f73f7f3c bf5000 bf8000 100073 f721217c 8048000 80f5000 1875 /bin/ f6cab90c 80f5000 80fa000 101873 /bin/ f724143c 80fa000 80ff000 100073 f68cf7ac 8574000 85aa000 100073 f6d354ec b7d81000 b7f81000 71 /usr/lib/locale/locale-archive f7594d84 b7f81000 b7f84000 100073 f6f24124 b7f84000 b7f85000 100073 f72e2d2c b7f85000 b7f8c000 d1 /usr/lib/gconv/gconv-modules.cache f72418b4 bfa8a000 bfa9f000 100173 crash> vtop 584000 VIRTUAL PHYSICAL 584000 37cd6000 PAGE DIRECTORY: f745c980 PGD: f745c980 => 369e0001 PMD: 369e0010 => 11b7c1067 PTE: 11b7c1c20 => 37cd6025 PAGE: 37cd6000 PTE PHYSICAL FLAGS 37cd6025 37cd6000 (PRESENT|USER|ACCESSED) VMA START END FLAGS FILE f6954d84 584000 585000 8000075 PAGE PHYSICAL MAPPING INDEX CNT FLAGS c16f9ac0 37cd6000 0 0 48 80000004 crash> Does the vtop command fall apart somewhere? BTW, if you haven't done it already, you should also take the dumpfile out of the picture, and just run crash on the live system. If by some stretch of the imagination *that* works, then you might have to point the finger back at kdump operation. In any case, at least you've got a situation where crash can at least deal with unity-mapped addresses. With those addresses it doesn't have to do any kind of page-table walk-throughs. I'm guessing that there's something in x86.c's x86_init() function in the PRE_GDB section that's not correct for your setup. There is support for Red Hat's older "hugemem" 4G/4G split kernels, where both the kernel and user space have 4G (over-lapping) virtual address regions, and so there may be some confusion there with yours. For starters, bring up the session as you did above, and enter "help -v" and "help -m". They're debug options that dump a couple internal crash data structures which may shed some light. Dave

Worth, Kevin wrote: Yeah, crash 4.0-3.7 is 2 years old, which is pretty ancient. Plus I'm only interested in helping out with the latest version. But according to the above, 4.0-7.2 works OK on the live system? You can do a "mod" command and it works OK? Sometimes on larger-memory systems, running live using /dev/mem, you might see the "WARNING: cannot access vmalloc'd module" message because the physical memory that is backing the vmalloc'd virtual address is in highmem, and cannot be accessed by /dev/mem. In any case, it appears that the module structures have all been read successfully on your live system. And that's kind of bothersome, because for all practical purposes, the crash utility doesn't care where it's getting the physical memory from (i.e., from /dev/mem or from the dumpfile). And if it works on the live system, it should work with the dumpfile. Anyway, looking at the crash.log, here's what's happening: Everything was running fine until the module initialization step. The list of installed kernel modules is headed up from the "modules" list_head symbol at 403c63a4, which contains a pointer to the first module structure at vmalloc address f9088280: ... <readmem: 403c63a4, KVADDR, "modules", 4, (FOE), 83ff8cc> please wait... (gathering module symbol data) module: f9088280 The readmem() of that first module -- and the very first vmalloc address -- at f9088280 required a page table translation: <readmem: f9088280, KVADDR, "module struct", 1536, (ROE|Q), 842a5e0> <readmem: 4044b000, KVADDR, "pgd page", 32, (FOE), 845a308> <readmem: 6000, PHYSADDR, "pmd page", 4096, (FOE), 845b310> <readmem: 1d515000, PHYSADDR, "page table", 4096, (FOE), 845c318> That readmem() appears to have worked, because it thinks it successfully read the module struct at that address. But when it pulled out the address of the *next* module in the linked list, it read this: module: fffffffc And when it tried to read that bogus address, it failed, and led to the WARNING message: <readmem: fffffffc, KVADDR, "module struct", 1536, (ROE|Q), 842a5e0> <readmem: 7000, PHYSADDR, "page table", 4096, (FOE), 845c318> crash: invalid kernel virtual address: fffffffc type: "module struct" WARNING: cannot access vmalloc'd module memory ... Although I cannot say for sure, I'm presuming that the initial read of the module structure at f9088280 ended up reading from the wrong location and therefore read garbage. You can verify that by bringing the a dumpfile session, and doing this: crash> module f9088280 It *should* display something that is recognizable as a module structure. For example: crash> mod | grep ext3 f8899080 ext3 123593 (not loaded) [CONFIG_KALLSYMS] crash> module f8899080 struct module { state = MODULE_STATE_LIVE, list = { next = 0xf8854a84, prev = 0xf8876984 }, name = "ext3" mkobj = { kobj = { k_name = 0xf88990cc "ext3", name = "ext3", kref = { refcount = { counter = 2 } }, ... Your attempt will probably show the fffffffc in the list_head just after the "state" field at the top, as well as a bunch of other garbage. And as I suggested in my first reply, can you also verify that user virtual address translations also fail? I suggested pulling a sample virtual address out of the current context's ("bash") VM, but doing that may "look" like it's working, but it may be doing it incorrectly. So you also need to verify the data that it finds there. One way to do that is to read the beginning of the /bin/bash text segment, and look for "ELF" string. For example, here I'm in a "bash" context, similar to the context that your dumpfile comes up in by default: crash> set PID: 19839 COMMAND: "bash" TASK: f7b03000 [THREAD_INFO: def66000] CPU: 1 STATE: TASK_INTERRUPTIBLE crash> Dump the virtual memory regions, and find the first VMA that is backed by "/bin/bash": crash> vm PID: 19839 TASK: f7b03000 CPU: 1 COMMAND: "bash" MM PGD RSS TOTAL_VM f6dc5740 f745c9c0 1392k 4532k VMA START END FLAGS FILE f69019bc 6fa000 703000 75 /lib/libnss_files-2.5.so f69013e4 703000 704000 100071 /lib/libnss_files-2.5.so f6901d84 704000 705000 100073 /lib/libnss_files-2.5.so f6901284 a7c000 a96000 875 /lib/ld-2.5.so f6901b74 a96000 a97000 100871 /lib/ld-2.5.so f6901b1c a97000 a98000 100873 /lib/ld-2.5.so f69012dc a9a000 bd7000 75 /lib/libc-2.5.so f690185c bd7000 bd9000 100071 /lib/libc-2.5.so f6901ac4 bd9000 bda000 100073 /lib/libc-2.5.so f69017ac bda000 bdd000 100073 f6901e8c bdf000 be1000 75 /lib/libdl-2.5.so f6901a6c be1000 be2000 100071 /lib/libdl-2.5.so f6901754 be2000 be3000 100073 /lib/libdl-2.5.so f6901f94 c89000 c8c000 75 /lib/libtermcap.so.2.0.8 f69016fc c8c000 c8d000 100073 /lib/libtermcap.so.2.0.8 f6901d2c fd1000 fd2000 8000075 f6901124 8047000 80f5000 1875 /bin/bash f69018b4 80f5000 80fa000 101873 /bin/bash f6901964 80fa000 80ff000 100073 f690122c 9a75000 9a96000 100073 f680890c b7d7f000 b7f7f000 71 /usr/lib/locale/locale-archive f6901f3c b7f7f000 b7f81000 100073 f68cfb74 b7f82000 b7f84000 100073 f6dd69bc b7f84000 b7f8b000 d1 /usr/lib/gconv/gconv-modules.cache f69014ec bf86e000 bf884000 100173 crash> You can see above, that in my case the text region starts at user virtual address 8047000. That actually points to the ELF header at the beginning of the "/bin/bash" file, which starts with a 0x7f followed by the ascii "ELF" characters: crash> rd 8047000 8047000: 464c457f .ELF crash> You might want to use "rd -u <address>" to ensure that crash will presume that the address is a user address, just in case that's an issue with your setup. Anyway, try the above, and also dump out the and save the output of these debug commands: crash> help -m > help.k crash> help -k > help.m crash> help -v > help.v But again, given that you seem to be saying that everything works just fine on the live system, the debugging of this issue will most likely end up requiring that you determine where exactly things "go wrong" with the dumpfile in comparison to the same things working correctly on the live system. Thanks, Dave

NOTE: I've restored the kexec list to this discussion because this 1G/3G issue does have ramifications w/respect to kexec-tools. I'm first going to ramble on about crash utility debugging for a bit here, but for the kexec/kdump masters in the audience, please at least take a look at the end of this message (do a "find in this message" for "KEXEC-KDUMP") where I discuss the kexec-tools hardwiring of the x86 PAGE_OFFSET to c000000, and whether it could screw up the dumpfile contents for Kevin's 1G/3G split where his PAGE_OFFSET is 40000000. First, the crash discussion... Worth, Kevin wrote: No, but it's 0x0, and therefore the "next" module in the list gets calculated as 0 - offset-of-list-member, or fffffffc. And "MODULE_STATE_LIVE" is being shown by dumb luck because its enumerator value is 0: ... Right, so we know bogus data is being read from the dumpfile. The question is: (1) whether the virtual-to-physical address translation is failing somehow, or (2) the dumpfile is screwed up. The fact that crash initially presumes that 8048000 and 80ee000 are kernel virtual addresses can be explained by this part of "help -v" debug output: flags: 515a (NODES_ONLINE|ZONES|PERCPU_KMALLOC_V2|COMMON_VADDR|KMEM_CACHE_INIT|FLATMEM|PERCPU_KMALLOC_V2_NODES) The "COMMON_VADDR" flag should *only* be set in the case of the Red Hat hugemem 4G/4G split kernel. However, I believe that crash should be able to continue even if the bit is set, as is the case when you run live. It is a crash issue having to do with your 4000000 PAGE_OFFSET, but I think it's benign, especially if user virtual address accesses run OK on your live system. That's one thing that needs verification. The "invalid user virtual address" messages above that you get *even* when you use "-u" would typically be generated as a result of the user virtual-to-physical address translation. However, they also could be generated if the virtual page being accessed has been swapped out. A better test would be translate all virtual address in the user address space in one fell swoop with "vm -p". It's a verbose command, but for each user virtual page in the current context, it will translate it to: (1) the current physical address location, or (2) if it's not in memory, but is backed by a file, it will show what file it comes from, or (3) if it's been swapped out, what swapfile location is has been swapped out to, or (4) if it's an anonymous page (with no file backing) that hasn't been touched yet, it will show "(not mapped)" Here's a truncated example: PID: 19839 TASK: f7b03000 CPU: 1 COMMAND: "bash" MM PGD RSS TOTAL_VM f6dc5740 f745c9c0 1392k 4532k VMA START END FLAGS FILE f69019bc 6fa000 703000 75 /lib/libnss_files-2.5.so VIRTUAL PHYSICAL 6fa000 12fdba000 6fb000 12fdbb000 6fc000 FILE: /lib/libnss_files-2.5.so OFFSET: 2000 6fd000 FILE: /lib/libnss_files-2.5.so OFFSET: 3000 6fe000 12f660000 6ff000 12f2cf000 700000 FILE: /lib/libnss_files-2.5.so OFFSET: 6000 701000 FILE: /lib/libnss_files-2.5.so OFFSET: 7000 702000 12fc6f000 VMA START END FLAGS FILE f69013e4 703000 704000 100071 /lib/libnss_files-2.5.so VIRTUAL PHYSICAL 703000 54791000 VMA START END FLAGS FILE f6901d84 704000 705000 100073 /lib/libnss_files-2.5.so VIRTUAL PHYSICAL 704000 12450d000 VMA START END FLAGS FILE f6901284 a7c000 a96000 875 /lib/ld-2.5.so VIRTUAL PHYSICAL a7c000 6ea28000 a7d000 101f62000 a7e000 6e6f3000 a7f000 6e07e000 a80000 6e084000 a81000 114c8e000 ... Run the command above on a "bash" context on *both* the live system and the dumpfile -- they should behave in a similar manner, but I'm guessing you may get some bizarre errors when you run it on the dumpfile. Getting back to the base problem with the bogus module read, here'a suggestion for debugging this. It requires that you run the live system, gather some basic data with the crash utility, and then enter "alt-sysrq-c". What we want to see is a virtual-to-physical translation of the first module in the module list on the live system. Then crash the system. Then we want to do the same thing on the subsequent vmcore to see if the same physical address references are made during the translation. So for example, on my live system, the "/dev/crash" kernel module is the last module entered, and therefore is pointed to by the base kernel's "modules" list_head: crash> p modules modules = $2 = { next = 0xf8bd0904, prev = 0xf882b104 } Subtract 4 from the "next" pointer, and display the module: crash> module 0xf8bd0900 struct module { state = MODULE_STATE_LIVE, list = { next = 0xf8caf984, prev = 0xc06787b0 }, name = "crash", mkobj = { kobj = { k_name = 0xf8bd094c "crash", name = "crash", kref = { refcount = { counter = 2 } }, ... Then translate it: crash> vtop 0xf8bd0900 VIRTUAL PHYSICAL f8bd0900 48ba1900 PAGE DIRECTORY: c0724000 PGD: c0724018 => 4001 PMD: 4e28 => 37ae067 PTE: 37aee80 => 48ba1163 PAGE: 48ba1000 PTE PHYSICAL FLAGS 48ba1163 48ba1000 (PRESENT|RW|ACCESSED|DIRTY|GLOBAL) PAGE PHYSICAL MAPPING INDEX CNT FLAGS c1917420 48ba1000 0 785045 1 c0000000 crash> Do the same type of thing on your live system (where you'll have a different module), and save the output in a file. Then immediately enter "alt-sysrq-c". With the resultant dumpfile, perform the same "p modules", "module <next-address-4>", and "vtop <next-address-4> steps as done above. The output *should* be identical, although we're primarily interested in the vtop output given that the "module <next-address-4>" will probably show garbage. (BTW, this presumes that the first module in the kernel list will still return bogus data like your current dumpfile. That may not be the case, and if so, we'll need to do something similar but different. For example, on the live system, capture the address of the "ext3" module, vtop it, crash the system, and then do the same thing in the dumpfile. You might want to do that anyway, just in case the default behavior is different. Then again, maybe it will work both live and in the dumpfile for the ext3 module address, in which case we'll need to go in a different debug-direction...) Show the outputs of the live system and the subsequent dumpfile. If they both end up resolving to the same physical address, then there's an issue with the dumpfile. KEXEC-KDUMP: I talked to Vivek Goyal, who originally wrote the kexec-tools facility, and he pointed me to this in the kexec-tools package's "kexec/arch/i386/crashdump-x86.h" file: #define PAGE_OFFSET 0xc0000000 #define __pa(x) ((unsigned long)(x)-PAGE_OFFSET) #define __VMALLOC_RESERVE (128 << 20) #define MAXMEM (-PAGE_OFFSET-__VMALLOC_RESERVE) where for x86, it hard-wires the x86 PAGE_OFFSET to c0000000, and will certainly result in a bogus MAXMEM given that your PAGE_OFFSET is 40000000. I don't know if that is related to the problem, but if you do a "readelf -a" of your vmcore file, you'll see some funky virtual address values for each PT_LOAD segment. They were dumped in the crash.log you sent me. Note that the virtual address regions (p_vaddr) are c0000000, c0100000, c5000000, ffffffffffffffff and ffffffffffffffff, all of which are incorrect or nonsensical w/respect to your 1G/3G split: Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 728 (2d8) p_vaddr: c0000000 p_paddr: 0 p_filesz: 655360 (a0000) p_memsz: 655360 (a0000) p_flags: 7 (PF_X|PF_W|PF_R) p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 656088 (a02d8) p_vaddr: c0100000 p_paddr: 100000 p_filesz: 15728640 (f00000) p_memsz: 15728640 (f00000) p_flags: 7 (PF_X|PF_W|PF_R) p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 16384728 (fa02d8) p_vaddr: c5000000 p_paddr: 5000000 p_filesz: 855638016 (33000000) p_memsz: 855638016 (33000000) p_flags: 7 (PF_X|PF_W|PF_R) p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 872022744 (33fa02d8) p_vaddr: ffffffffffffffff p_paddr: 38000000 p_filesz: 2272854016 (87790000) p_memsz: 2272854016 (87790000) p_flags: 7 (PF_X|PF_W|PF_R) p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 3144876760 (bb7302d8) p_vaddr: ffffffffffffffff p_paddr: 100000000 p_filesz: 1073741824 (40000000) p_memsz: 1073741824 (40000000) p_flags: 7 (PF_X|PF_W|PF_R) p_align: 0 Now, the crash utility only uses the p_paddr physical address fields for x86 dumpfiles, so that shouldn't be a problem. But I wonder whether when the /proc/vmcore is put together that there isn't some problem with the data that it accesses? Thanks, Dave

Whoops! Important typo.... "other than that it involves network traffic and should NOT be causing any funky behavior here" -Kevin -----Original Message----- From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf Of Worth, Kevin Sent: Saturday, October 04, 2008 10:35 AM To: Discussion list for crash utility usage, maintenance and development; kexec-ml Subject: RE: [Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash Hi Dave (and kexec list), Including kexec list in this email because Dave mentioned: "Show the outputs of the live system and the subsequent dumpfile. If they both end up resolving to the same physical address, then there's an issue with the dumpfile." Things appear to be resolving to the same address (though I suspect Dave can confirm). Please see below. I did have to censor one of the lines a bit- I do have a proprietary kernel module that I'm not able to discuss much about other than that it involves network traffic and should be causing any funky behavior here (the ext3 module seems to behave the same way). I just changed its name to "custom_lkm". One additional note, although my "running system" kernel has the modified 3G kernel / 1G user split, my "capture kernel" is just the standard Ubuntu kernel (with 1G kernel / 3G user). The system panic'ed immediately if I tried to use my modified kernel as the "capture kernel". I figured this was outside the norm so I've been using the standard kernel to perform the capture. First I ran through some commands Dave suggested on the live system (my contexts for the live system and dump were different, but what might be more important is that "vm -p" on a live system produced errors, while on the dump it did not): crash> vm PID: 32227 TASK: 47bc8030 CPU: 0 COMMAND: "crash" MM PGD RSS TOTAL_VM f7e67040 5fddfe00 63336k 67412k VMA START END FLAGS FILE f3ed61d4 8048000 83e5000 1875 /root/crash f3ed6d84 83e5000 83fc000 101877 /root/crash .... crash> vm -p PID: 32227 TASK: 47bc8030 CPU: 0 COMMAND: "crash" MM PGD RSS TOTAL_VM f7e67040 5fddfe00 63336k 67412k VMA START END FLAGS FILE f3ed61d4 8048000 83e5000 1875 /root/crash VIRTUAL PHYSICAL vm: read error: physical address: 10b60b000 type: "page table" crash> p modules modules = $2 = { next = 0xf9088284, prev = 0xf8842104 } crash> module 0xf9088280 struct module { state = MODULE_STATE_LIVE, list = { next = 0xf8ff9d84, prev = 0x403c63a4 }, name = "custom_lkm\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00 0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00 0\000\000\000\000\000\000\000\000\000\000\000\000\000", mkobj = { kobj = { k_name = 0xf90882cc "custom_lkm", name = "custom_lkm\000\000\000\000\000\000\000\000", kref = { refcount = { counter = 3 } }, entry = { next = 0x403c6068, prev = 0xf8ff9de4 }, ... crash> vtop 0xf9088280 VIRTUAL PHYSICAL f9088280 119b98280 PAGE DIRECTORY: 4044b000 PGD: 4044b018 => 6001 PMD: 6e40 => 1d515067 PTE: 1d515440 => 119b98163 PAGE: 119b98000 PTE PHYSICAL FLAGS 119b98163 119b98000 (PRESENT|RW|ACCESSED|DIRTY|GLOBAL) crash> mod | grep ext3 f88c8000 ext3 132616 (not loaded) [CONFIG_KALLSYMS] crash> module 0xf88c8000 struct module { state = MODULE_STATE_LIVE, list = { next = 0xf88a6604, prev = 0xf885d584 }, name = "ext3\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 00\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 00\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000", mkobj = { kobj = { k_name = 0xf88c804c "ext3", name = "ext3\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000", kref = { refcount = { counter = 3 } }, entry = { next = 0xf885d5e4, prev = 0xf88a6664 }, ... (Realized afterward that I forgot to vtop ext3. Let me know if it's needed and I can repeat this procedure) crash> vm PID: 4323 TASK: 47be0a90 CPU: 0 COMMAND: "bash" MM PGD RSS TOTAL_VM 5d683580 5d500dc0 2616k 3968k VMA START END FLAGS FILE 5fc2aac4 8048000 80ee000 1875 /bin/bash 5fe5f0cc 80ee000 80f3000 101877 /bin/bash ... crash> vm -p PID: 4323 TASK: 47be0a90 CPU: 0 COMMAND: "bash" MM PGD RSS TOTAL_VM 5d683580 5d500dc0 2616k 3968k VMA START END FLAGS FILE 5fc2aac4 8048000 80ee000 1875 /bin/bash VIRTUAL PHYSICAL 8048000 FILE: /bin/bash OFFSET: 0 8049000 FILE: /bin/bash OFFSET: 1000 804a000 FILE: /bin/bash OFFSET: 2000 ...no errors, lots of output crash> modules modules = $2 = { next = 0xf9088284, prev = 0xf8842104 } crash> module 0xf9088280 struct module { state = MODULE_STATE_LIVE, list = { next = 0x0, prev = 0x0 }, name = "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 00\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 00\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 00\000", mkobj = { kobj = { k_name = 0x0, name = "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 00\000\000", kref = { refcount = { counter = 0 } }, entry = { next = 0x0, prev = 0x0 ... crash> vtop 0xf9088280 VIRTUAL PHYSICAL f9088280 119b98280 PAGE DIRECTORY: 4044b000 PGD: 4044b018 => 6001 PMD: 6e40 => 1d515067 PTE: 1d515440 => 119b98163 PAGE: 119b98000 PTE PHYSICAL FLAGS 119b98163 119b98000 (PRESENT|RW|ACCESSED|DIRTY|GLOBAL) PAGE PHYSICAL MAPPING INDEX CNT FLAGS 47337300 119b98000 0 0 1 80000000 crash> mod | grep ext3 mod: cannot access vmalloc'd module memory (using the same address that ext3 had in a running system) crash> module 0xf88c8000 struct module { state = MODULE_STATE_LIVE, list = { next = 0x0, prev = 0x0 }, name = "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 00\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 00\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 00\000", mkobj = { kobj = { k_name = 0x0, name = "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 00\000\000", kref = { refcount = { counter = 0 } }, entry = { next = 0x0, prev = 0x0 ... crash> vtop 0xf88c8000 VIRTUAL PHYSICAL f88c8000 13905f000 PAGE DIRECTORY: 4044b000 PGD: 4044b018 => 6001 PMD: 6e20 => 1d5fc067 PTE: 1d5fc640 => 13905f163 PAGE: 13905f000 PTE PHYSICAL FLAGS 13905f163 13905f000 (PRESENT|RW|ACCESSED|DIRTY|GLOBAL) PAGE PHYSICAL MAPPING INDEX CNT FLAGS Thanks again for your continued assistance. I hope this is helpful information. -Kevin -----Original Message----- From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson Sent: Friday, October 03, 2008 8:44 AM To: Discussion list for crash utility usage, maintenance and development; kexec(a)lists.infradead.org Subject: Re: [Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash NOTE: I've restored the kexec list to this discussion because this 1G/3G issue does have ramifications w/respect to kexec-tools. I'm first going to ramble on about crash utility debugging for a bit here, but for the kexec/kdump masters in the audience, please at least take a look at the end of this message (do a "find in this message" for "KEXEC-KDUMP") where I discuss the kexec-tools hardwiring of the x86 PAGE_OFFSET to c000000, and whether it could screw up the dumpfile contents for Kevin's 1G/3G split where his PAGE_OFFSET is 40000000. First, the crash discussion... Worth, Kevin wrote: No, but it's 0x0, and therefore the "next" module in the list gets calculated as 0 - offset-of-list-member, or fffffffc. And "MODULE_STATE_LIVE" is being shown by dumb luck because its enumerator value is 0: ... Right, so we know bogus data is being read from the dumpfile. The question is: (1) whether the virtual-to-physical address translation is failing somehow, or (2) the dumpfile is screwed up. The fact that crash initially presumes that 8048000 and 80ee000 are kernel virtual addresses can be explained by this part of "help -v" debug output: flags: 515a (NODES_ONLINE|ZONES|PERCPU_KMALLOC_V2|COMMON_VADDR|KMEM_CACHE_INIT|FLATMEM|PERCPU_KMALLOC_V2_NODES) The "COMMON_VADDR" flag should *only* be set in the case of the Red Hat hugemem 4G/4G split kernel. However, I believe that crash should be able to continue even if the bit is set, as is the case when you run live. It is a crash issue having to do with your 4000000 PAGE_OFFSET, but I think it's benign, especially if user virtual address accesses run OK on your live system. That's one thing that needs verification. The "invalid user virtual address" messages above that you get *even* when you use "-u" would typically be generated as a result of the user virtual-to-physical address translation. However, they also could be generated if the virtual page being accessed has been swapped out. A better test would be translate all virtual address in the user address space in one fell swoop with "vm -p". It's a verbose command, but for each user virtual page in the current context, it will translate it to: (1) the current physical address location, or (2) if it's not in memory, but is backed by a file, it will show what file it comes from, or (3) if it's been swapped out, what swapfile location is has been swapped out to, or (4) if it's an anonymous page (with no file backing) that hasn't been touched yet, it will show "(not mapped)" Here's a truncated example: PID: 19839 TASK: f7b03000 CPU: 1 COMMAND: "bash" MM PGD RSS TOTAL_VM f6dc5740 f745c9c0 1392k 4532k VMA START END FLAGS FILE f69019bc 6fa000 703000 75 /lib/libnss_files-2.5.so VIRTUAL PHYSICAL 6fa000 12fdba000 6fb000 12fdbb000 6fc000 FILE: /lib/libnss_files-2.5.so OFFSET: 2000 6fd000 FILE: /lib/libnss_files-2.5.so OFFSET: 3000 6fe000 12f660000 6ff000 12f2cf000 700000 FILE: /lib/libnss_files-2.5.so OFFSET: 6000 701000 FILE: /lib/libnss_files-2.5.so OFFSET: 7000 702000 12fc6f000 VMA START END FLAGS FILE f69013e4 703000 704000 100071 /lib/libnss_files-2.5.so VIRTUAL PHYSICAL 703000 54791000 VMA START END FLAGS FILE f6901d84 704000 705000 100073 /lib/libnss_files-2.5.so VIRTUAL PHYSICAL 704000 12450d000 VMA START END FLAGS FILE f6901284 a7c000 a96000 875 /lib/ld-2.5.so VIRTUAL PHYSICAL a7c000 6ea28000 a7d000 101f62000 a7e000 6e6f3000 a7f000 6e07e000 a80000 6e084000 a81000 114c8e000 ... Run the command above on a "bash" context on *both* the live system and the dumpfile -- they should behave in a similar manner, but I'm guessing you may get some bizarre errors when you run it on the dumpfile. Getting back to the base problem with the bogus module read, here'a suggestion for debugging this. It requires that you run the live system, gather some basic data with the crash utility, and then enter "alt-sysrq-c". What we want to see is a virtual-to-physical translation of the first module in the module list on the live system. Then crash the system. Then we want to do the same thing on the subsequent vmcore to see if the same physical address references are made during the translation. So for example, on my live system, the "/dev/crash" kernel module is the last module entered, and therefore is pointed to by the base kernel's "modules" list_head: crash> p modules modules = $2 = { next = 0xf8bd0904, prev = 0xf882b104 } Subtract 4 from the "next" pointer, and display the module: crash> module 0xf8bd0900 struct module { state = MODULE_STATE_LIVE, list = { next = 0xf8caf984, prev = 0xc06787b0 }, name = "crash", mkobj = { kobj = { k_name = 0xf8bd094c "crash", name = "crash", kref = { refcount = { counter = 2 } }, ... Then translate it: crash> vtop 0xf8bd0900 VIRTUAL PHYSICAL f8bd0900 48ba1900 PAGE DIRECTORY: c0724000 PGD: c0724018 => 4001 PMD: 4e28 => 37ae067 PTE: 37aee80 => 48ba1163 PAGE: 48ba1000 PTE PHYSICAL FLAGS 48ba1163 48ba1000 (PRESENT|RW|ACCESSED|DIRTY|GLOBAL) PAGE PHYSICAL MAPPING INDEX CNT FLAGS c1917420 48ba1000 0 785045 1 c0000000 crash> Do the same type of thing on your live system (where you'll have a different module), and save the output in a file. Then immediately enter "alt-sysrq-c". With the resultant dumpfile, perform the same "p modules", "module <next-address-4>", and "vtop <next-address-4> steps as done above. The output *should* be identical, although we're primarily interested in the vtop output given that the "module <next-address-4>" will probably show garbage. (BTW, this presumes that the first module in the kernel list will still return bogus data like your current dumpfile. That may not be the case, and if so, we'll need to do something similar but different. For example, on the live system, capture the address of the "ext3" module, vtop it, crash the system, and then do the same thing in the dumpfile. You might want to do that anyway, just in case the default behavior is different. Then again, maybe it will work both live and in the dumpfile for the ext3 module address, in which case we'll need to go in a different debug-direction...) Show the outputs of the live system and the subsequent dumpfile. If they both end up resolving to the same physical address, then there's an issue with the dumpfile. KEXEC-KDUMP: I talked to Vivek Goyal, who originally wrote the kexec-tools facility, and he pointed me to this in the kexec-tools package's "kexec/arch/i386/crashdump-x86.h" file: #define PAGE_OFFSET 0xc0000000 #define __pa(x) ((unsigned long)(x)-PAGE_OFFSET) #define __VMALLOC_RESERVE (128 << 20) #define MAXMEM (-PAGE_OFFSET-__VMALLOC_RESERVE) where for x86, it hard-wires the x86 PAGE_OFFSET to c0000000, and will certainly result in a bogus MAXMEM given that your PAGE_OFFSET is 40000000. I don't know if that is related to the problem, but if you do a "readelf -a" of your vmcore file, you'll see some funky virtual address values for each PT_LOAD segment. They were dumped in the crash.log you sent me. Note that the virtual address regions (p_vaddr) are c0000000, c0100000, c5000000, ffffffffffffffff and ffffffffffffffff, all of which are incorrect or nonsensical w/respect to your 1G/3G split: Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 728 (2d8) p_vaddr: c0000000 p_paddr: 0 p_filesz: 655360 (a0000) p_memsz: 655360 (a0000) p_flags: 7 (PF_X|PF_W|PF_R) p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 656088 (a02d8) p_vaddr: c0100000 p_paddr: 100000 p_filesz: 15728640 (f00000) p_memsz: 15728640 (f00000) p_flags: 7 (PF_X|PF_W|PF_R) p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 16384728 (fa02d8) p_vaddr: c5000000 p_paddr: 5000000 p_filesz: 855638016 (33000000) p_memsz: 855638016 (33000000) p_flags: 7 (PF_X|PF_W|PF_R) p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 872022744 (33fa02d8) p_vaddr: ffffffffffffffff p_paddr: 38000000 p_filesz: 2272854016 (87790000) p_memsz: 2272854016 (87790000) p_flags: 7 (PF_X|PF_W|PF_R) p_align: 0 Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 3144876760 (bb7302d8) p_vaddr: ffffffffffffffff p_paddr: 100000000 p_filesz: 1073741824 (40000000) p_memsz: 1073741824 (40000000) p_flags: 7 (PF_X|PF_W|PF_R) p_align: 0 Now, the crash utility only uses the p_paddr physical address fields for x86 dumpfiles, so that shouldn't be a problem. But I wonder whether when the /proc/vmcore is put together that there isn't some problem with the data that it accesses? Thanks, Dave -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

Dave, That does seem pretty strange that the physical address is coming out beyond the 4GB mark and that the read actually succeeds. Just checked on the Ubuntu patches to the 2.6.20 kernel ( http://archive.ubuntu.com/ubuntu/pool/main/l/linux-source-2.6.20/linux-so... ) and no mention of mem.c or either of those two functions. Let me try the kexec PAGE_OFFSET modification today or tomorrow and reply back on how it goes. If that produces no change I'll try do a re-run of the previous email's process with some more careful attention paid (that I get a vtop of everything and that my context examples are the same process). -Kevin ________________________________________ From: crash-utility-bounces(a)redhat.com [crash-utility-bounces(a)redhat.com] On Behalf Of Dave Anderson [anderson(a)redhat.com] Sent: Monday, October 06, 2008 8:10 AM To: Discussion list for crash utility usage, maintenance and development Cc: kexec-ml Subject: Re: [Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash ----- "Kevin Worth" <kevin.worth(a)hp.com&gt; wrote: OK, let's skip the user-space angle for now, because I keep forgetting that you are running with /dev/mem as the memory source. And there is an inconsistency with your debug output that I cannot explain. As I mentioned before, the /dev/mem driver has this immediate restriction in "drivers/char/mem.c": static ssize_t read_mem(struct file * file, char __user * buf, size_t count, loff_t *ppos) { unsigned long p = *ppos; ssize_t read, sz; char *ptr; if (!valid_phys_addr_range(p, count)) return -EFAULT; ... where for x86, it looks like this: static inline int valid_phys_addr_range(unsigned long addr, size_t count) { if (addr + count > __pa(high_memory)) return 0; return 1; } That restricts is from reading "highmem", which is the extent of physical memory that can be unity-mapped, which means that the kernel can directly access it by simply adding the PAGE_OFFSET value to the physical address. In your case, your PAGE_OFFSET is 0x40000000. With your 1G/3G split, you've got 3GB of kernel virtual address space that you can directly access, minus 128MB at the top that is used for the vmalloc() address range. (3GB - 128MB) is 0xb8000000. Therefore, your "high_memory" maximum unity-mapped kernel virtual address is (0xb8000000 + PAGE_OFFSET), or in your case is 0xf8000000, your high_memory value is 0xf8000000. In any case, on your live system, whenever a crash utility readmem() is done that accesses a physical address beyond 0xb8000000, it *should* get back the EFAULT above and fail, and therefore the crash command making the readmem() fails. Accordingly, when you did this on your live system: It ended up translating the first user virtual address (8048000), requiring a page-table translation, and ended up trying to access a page table page at physical address 0x10b60b000, which /dev/mem did not allow, because you got a "read error". However -- and this is what I cannot explain -- the above can also happen on a live system when accessing vmalloc() kernel virtual space as well *if* any PTE or page table read to make the translation, or *if* the ending physical page itself, are beyond the /dev/mem restriction (again, which should be 0xb8000000 in your case). So when you did this on your live system, you referenced the vmalloc address of your custom module at address 0xf9088280, and successfully read and displayed its contents: But when you did vtop of 0xf9088280, it ended up translating to 119b98000, which is well beyond 4GB (never mind 0xb8000000), so /dev/mem should not have been able to read it: By any chance has the /dev/mem driver been modified on your kernel? In any case, I can't explain why you are apprently able to access physical addresses beyond your "high_memory"? an. Anyway, the ext3 translation is useless without the accompanying "vtop": And the "bash" vm output only makes sense with respect to its output on the live system: But getting back to vmalloc'd module space, your access of the module at vmalloc-address-f9088280/physical-address-119b98000 showed that it's getting back a page of zeroes, while accessing the same physical address (0x119b98000) the you successfully read (but how?) on the live system: > > crash> modules > modules = $2 = { > next = 0xf9088284, > prev = 0xf8842104 > } > > crash> module 0xf9088280 > struct module { > state = MODULE_STATE_LIVE, > list = { > next = 0x0, > prev = 0x0 > }, > name = "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 > 00\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 > 00\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 > 00\000", > mkobj = { > kobj = { > k_name = 0x0, > name = "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 > 00\000\000", > kref = { > refcount = { > counter = 0 > } > }, > entry = { > next = 0x0, > prev = 0x0 > ... > PAGE PHYSICAL MAPPING INDEX CNT FLAGS > 47337300 119b98000 0 0 1 80000000 And so even though I'd like to point out that analogous readmem() on the dumpfile reads the same physical location -- and seems to just return zeroes -- is not enough for me to simply state that it's a problem with kexec/kdump. Because, again, I cannot explain how you are able to access physical address 0x119b98000 from /dev/mem on your live system? Can you check whether your kernel source has modified the read_mem() or valid_phys_addr_range() functions? If they unchanged from what I showed above (from 2.6.20), then I'm stumped, because it makes no sense to me how you can read from those physical addresses on your live system. For verification, if you do this: crash> p high_memory it should show 0xf8000000. If you then do a vtop of 0xf8000000, it will simply end up stripping off the PAGE_OFFSET of 0x40000000, resulting in the maximum-accessible physical address of 0xb8000000. And if you can do this: crash> rd -p 0xb8000000 it should fail -- as should any address equal to or above it. But your output above that translates the module vmalloc addresses seemingly reads physical addresses well beyond the 4GB (0x100000000). And that's what I cannot begin to explain. So I'm running out of ideas here... One thing I can suggest is to rebuild your kexec-tools package that you're using, and correct the PAGE_OFFSET value to equal your system's. The version of "kexec/arch/i386/crashdump-x86.h" that we (Red Hat) are using looks like this: #ifndef CRASHDUMP_X86_H #define CRASHDUMP_X86_H struct kexec_info; int load_crashdump_segments(struct kexec_info *info, char *mod_cmdline, unsigned long max_addr, unsigned long min_base); #define PAGE_OFFSET 0xc0000000 #define __pa(x) ((unsigned long)(x)-PAGE_OFFSET) #define __VMALLOC_RESERVE (128 << 20) #define MAXMEM (-PAGE_OFFSET-__VMALLOC_RESERVE) #define CRASH_MAX_MEMMAP_NR (KEXEC_MAX_SEGMENTS + 1) #define CRASH_MAX_MEMORY_RANGES (MAX_MEMORY_RANGES + 2) /* Backup Region, First 640K of System RAM. */ #define BACKUP_SRC_START 0x00000000 #define BACKUP_SRC_END 0x0009ffff #define BACKUP_SRC_SIZE (BACKUP_SRC_END - BACKUP_SRC_START + 1) #endif /* CRASHDUMP_X86_H */ Try rebuilding your package with PAGE_OFFSET defined as 0x40000000, and then see what happens. Dave -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility