Re: [Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash

Tried version 2.0 of kexec-tools (released 7/19/2008) and still have the same problem with zero'ed out module info. Sounds like perhaps it is narrowed down to the 2.6.20 kernel kdump code (the most difficult part to change out, but the part that likely gets a lot of eyes on it and probably has the issue fixed in current versions). :\

-Kevin -----Original Message----- From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf Of Worth, Kevin Sent: Wednesday, October 15, 2008 2:43 PM To: Discussion list for crash utility usage, maintenance and development Subject: RE: [Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash Sorry please ignore the second paragraph... I am already running the most recent version in Ubuntu, 20070330 from Simon Horman's kexec-tools-testing and kernel 2.6.20. ... may try a newer version of kexec-tools-testing to see if anything changes. -Kevin -----Original Message----- From: Worth, Kevin Sent: Wednesday, October 15, 2008 2:31 PM To: Discussion list for crash utility usage, maintenance and development Subject: RE: [Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash So Dave, at this am I correct in the assumption that it sounds like this is not a problem with crash, but with the dump file itself? I tried one more go at modifying the kexec-tools to have the correct PAGE_OFFSET defined and still got the same type of results (all zeroes at the module's address), so that doesn't seem to be it. Maybe this is a better question to take to the kexec mailing list, but do you know where the line is drawn between the kernel support or the userspace (kexec-tools)? I'm presuming that the kernel support is tied to each kernel (i.e. since I'm on 2.6.20 this issue could have been resolved in a more recent kernel). I'm wondering if I can pull a newer kexec-tools and that they might work with 2.6.20 and possibly have this issue resolved. -Kevin -----Original Message----- From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson Sent: Wednesday, October 15, 2008 6:53 AM To: Discussion list for crash utility usage, maintenance and development Subject: Re: [Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash ----- "Kevin Worth" <kevin.worth(a)hp.com&gt; wrote: > Hi Dave, > > Before you responded I noticed that a simple "make modules" didn't > work because my kernel wasn't exporting the symbol. Rather than do > anything risky/complex which might risk mucking up the troubleshooting > process, I just rebuilt the kernel. It built just fine and now I can > load crash and I see "DUMPFILE: /dev/crash" when I load up crash. Let > me try walking through the steps that you had me do previously, this > time using /dev/crash instead of /dev/mem and /dev/kmem You made one small error (but not totally fatal) in the suggested steps. See my comments below... > > >From my limited understanding of what's going on here, it would > appear that the dump file is missing some data, or else crash is > looking in the wrong place for it. The crash utility is a slave to what is indicated in the PT_LOAD segments of the ELF header of the kdump vmcore. In the case of the physical memory chunk that starts at 4GB physical on your machine, this is what's in the ELF header (from your original "crash.log" file): Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 3144876760 (bb7302d8) p_vaddr: ffffffffffffffff p_paddr: 100000000 p_filesz: 1073741824 (40000000) p_memsz: 1073741824 (40000000) p_flags: 7 (PF_X|PF_W|PF_R) p_align: 0 What that says is: for the range of physical memory starting at 0x100000000 (p_paddr), the vmcore contains a block of memory starting at file offset (p_offset) 3144876760/0xbb7302d8 that is 1073741824/0x40000000 (p_filesz) bytes long. More simply put, the 1GB of physical memory from 4GB to 5GB can be found in the vmcore file starting at file offset 3144876760. So if a request for physical memory page 0x100000000 comes in, the crash utility reads from vmcore file offset 3144876760. If the next physical page were requested, i.e., at 0x100001000, it would read from vmcore file offset 3144876760+4096. It's as simple as that -- so when you suggest that "crash is looking in the wrong place for it", well, there's nothing that the crash utility can do differently. Now, back to the test sequence: > ---Live system--- > > KERNEL: vmlinux-devcrash > DUMPFILE: /dev/crash > CPUS: 2 > DATE: Tue Oct 14 16:08:28 2008 > UPTIME: 00:02:07 > LOAD AVERAGE: 0.17, 0.08, 0.03 > TASKS: 97 > NODENAME: test-machine > RELEASE: 2.6.20-17.39-custom2 > VERSION: #1 SMP Tue Oct 14 13:45:17 PDT 2008 > MACHINE: i686 (2200 Mhz) > MEMORY: 5 GB > PID: 5628 > COMMAND: "crash" > TASK: 5d4c2560 [THREAD_INFO: f3de6000] > CPU: 1 > STATE: TASK_RUNNING (ACTIVE) > > crash> p modules > modules = $2 = { > next = 0xf8a3ea04, > prev = 0xf8842104 > } > > crash> module 0xf8a3ea00 > struct module { > state = MODULE_STATE_LIVE, > list = { > next = 0xf8d10484, > prev = 0x403c63a4 > }, > name = > "crash\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\ > 000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\ > 000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000", > mkobj = { > kobj = { > k_name = 0xf8a3ea4c "crash", > name = > "crash\000\000\000\000\000\000\000\000\000\000\000\000\000\000", > kref = { > refcount = { > counter = 3 > } > }, > entry = { > next = 0x403c6068, > prev = 0xf8d104e4 > }, > parent = 0x403c6074 > ... > > crash> vtop 0xf8a3ea00 > VIRTUAL PHYSICAL > f8a3ea00 116017a00 OK -- so the physical memory location of the module data structure is at physical address 116017a00, but... > > PAGE DIRECTORY: 4044b000 > PGD: 4044b018 => 6001 > PMD: 6e28 => 1d51a067 > PTE: 1d51a1f0 => 116017163 > PAGE: 116017000 > > PTE PHYSICAL FLAGS > 116017163 116017000 (PRESENT|RW|ACCESSED|DIRTY|GLOBAL) > > PAGE PHYSICAL MAPPING INDEX CNT FLAGS > 472c02e0 116017000 0 229173 1 80000000 > You're reading from the beginning of the page, i.e., 116017000 instead of where the module structure is at 116017a00: > crash> rd -p 116017000 30 > 116017000: 53e58955 d089c389 4d8bca89 74c98508 U..S.......M...t > 116017010: 01e9831f b85b0d74 ffffffea ffffba5d ....t.[.....]... > 116017020: 03c3ffff 53132043 26b48d24 00000000 ....C .S$..&.... > 116017030: 89204389 5d5b2453 26b48dc3 00000000 .C .S$[]...&.... > 116017040: 83e58955 55892cec 08558be4 89f45d89 U....,.U..U..].. > 116017050: 7d89f875 ffeabffc 4d89ffff 8b028be0 u..}.......M.... > 116017060: c3890452 ac0fd689 45890cf3 0ceec1ec R..........E.... > 116017070: 5589c889 89d231f0 ...U.1.. > crash> > So therefore you're not seeing the "crash" strings embedded in the raw physical data. Now, although it would have been "nice" if you could have shown the contents of the module structure via the physical address, the fact remains that since you used the /dev/crash driver, the "module 0xf8a3ea00" command required that the crash utility first translate the vmalloc address into its physical equivalent, and then read from there. In any case, you do have a dump of physical memory from 116017000 which at least is in the same 4k page as the module data structure, so it should not change when read from the dumpfile. > ---Using dump file--- > > > please wait... (gathering module symbol data) > WARNING: cannot access vmalloc'd module memory > > KERNEL: vmlinux-devcrash > DUMPFILE: /var/crash/vmcore > CPUS: 2 > DATE: Tue Oct 14 16:09:32 2008 > UPTIME: 00:03:12 > LOAD AVERAGE: 0.09, 0.08, 0.02 > TASKS: 97 > NODENAME: test-machine > RELEASE: 2.6.20-17.39-custom2 > VERSION: #1 SMP Tue Oct 14 13:45:17 PDT 2008 > MACHINE: i686 (2200 Mhz) > MEMORY: 5 GB > PANIC: "[ 192.148000] SysRq : Trigger a crashdump" > PID: 0 > COMMAND: "swapper" > TASK: 403c0440 (1 of 2) [THREAD_INFO: 403f2000] > CPU: 0 > STATE: TASK_RUNNING (SYSRQ) > > crash> p modules > modules = $2 = { > next = 0xf8a3ea04, > prev = 0xf8842104 > } > > crash> module 0xf8a3ea00 > struct module { > state = MODULE_STATE_LIVE, > list = { > next = 0x0, > prev = 0x0 > }, > name = > "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 > 00\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 > 00\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 > 00\000", > mkobj = { > kobj = { > k_name = 0x0, > name = > "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0 > 00\000\000", > kref = { > refcount = { > counter = 0 > } > }, > entry = { > next = 0x0, > prev = 0x0 > ... > > crash> vtop 0xf8a3ea00 > VIRTUAL PHYSICAL > f8a3ea00 116017a00 > > PAGE DIRECTORY: 4044b000 > PGD: 4044b018 => 6001 > PMD: 6e28 => 1d51a067 > PTE: 1d51a1f0 => 116017163 > PAGE: 116017000 > > PTE PHYSICAL FLAGS > 116017163 116017000 (PRESENT|RW|ACCESSED|DIRTY|GLOBAL) > > PAGE PHYSICAL MAPPING INDEX CNT FLAGS > 472c02e0 116017000 0 229173 1 80000000 > > crash> rd -p 116017000 30 > 116017000: 00000000 00000000 00000000 00000000 ................ > 116017010: 00000000 00000000 00000000 00000000 ................ > 116017020: 00000000 00000000 00000000 00000000 ................ > 116017030: 00000000 00000000 00000000 00000000 ................ > 116017040: 00000000 00000000 00000000 00000000 ................ > 116017050: 00000000 00000000 00000000 00000000 ................ > 116017060: 00000000 00000000 00000000 00000000 ................ > 116017070: 00000000 00000000 ........ > crash> Now we're reading the same physical address as you did on the dumpfile, and it's returning all zeroes. And the "module 0xf8a3ea00" above shows all zeroes from a higher location in the page because the same vmalloc translation is done to turn it into a physical address before reading it from the vmcore file. But instead of using the /dev/crash driver to access the translated physical memory, the crash utility uses the information from the ELF header's PT_LOAD segments to find out where to find the page data in the vmcore file. So, anyway, the "rd -p 116017000 30" command that you did on both the live system and the dumpfile should yield the same data. It seems like in all examples to date, the file data read at the greater-than-4GB PT_LOAD segment returns zeroes. You can verify this from the crash utility's viewpoint by doing a "help -n" during runtime when running with the dumpfile, which will show you both the actual contents of the ELF header, as well as the manner in which the PT_LOAD data is stored for its use. (It's also shown with the "crash -d7 ..." output). So again, from your original "crash.log" file, here is what the ELF header's PT_LOAD segment contains: Elf64_Phdr: p_type: 1 (PT_LOAD) p_offset: 3144876760 (bb7302d8) p_vaddr: ffffffffffffffff p_paddr: 100000000 p_filesz: 1073741824 (40000000) p_memsz: 1073741824 (40000000) p_flags: 7 (PF_X|PF_W|PF_R) p_align: 0 And this is what the crash utility stored in its internal data structure for that particular segment: pt_load_segment[4]: file_offset: bb7302d8 phys_start: 100000000 phys_end: 140000000 zero_fill: 0 And when the physical memory read request comes in, it filters to this part of the crash utility's read_netdump() function in netdump.c: for (i = offset = 0; i < nd->num_pt_load_segments; i++) { pls = &nd->pt_load_segments[i]; if ((paddr >= pls->phys_start) && (paddr < pls->phys_end)) { offset = (off_t)(paddr - pls->phys_start) + pls->file_offset; break; } if (pls->zero_fill && (paddr >= pls->phys_end) && (paddr < pls->zero_fill)) { memset(bufptr, 0, cnt); return cnt; } } So for any physical address request between 100000000 to 140000000, (4GB to 5GB) it will calculate the offset to seek to by subtracting 100000000 from the incoming physical address, and adding the difference to the starting file offset of the whole segment. So if you wanted to, you could put debug code just prior to the "break" above that shows the pls->file_offset for a given incoming physical address. But this code has been in place forever, so it's hard to conceive that somehow it's not working in the case of this dumpfile. But presuming that it *does* go to the correct file offset location in the vmcore, and it's getting bogus data from there, then there's nothing that the crash utility can do about it. Dave -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility