Re: [Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash
----- "Kevin Worth" <kevin.worth(a)hp.com> wrote:
Hi Dave,
I tried changing the PAGE_OFFSET definition in kexec-tools. Didn't
seem to affect it- crash still fails to load the vmalloc'ed memory. If
that seems like it absolves kexec-tools of any sins then perhaps we
can drop the kexec-ml off the CC list.
Your statement "Theoretically, anything at and above 0xb8000000 should
fail." was accurate, which I saw on my live system (with no dump
involved). Hoping this provides some insight.
Right -- but when you did the "module 0xf9102280", it read legitimate
data -- but the translated vmalloc address of 0xf9102280 shows a physical
address of 119b76280, i.e. well beyond the physical limit of 0xb8000000:
crash> module 0xf9102280
struct module {
state = MODULE_STATE_LIVE,
list = {
next = 0xf9073d84,
prev = 0x403c63a4
},
name = "custom_lkm"
...
crash> vtop 0xf9102280
VIRTUAL PHYSICAL
f9102280 119b76280
...
crash> rd -p 119b76000 30
rd: read error: physical address: 119b76000 type: "32-bit PHYSADDR"
...
That being the case, I just remembered something that I had completely
forgotten about -- because of yet another Red Hat imposed restriction.
On RHEL systems, we have the restricted /dev/mem, but in addition to
that /dev/kmem has been completely removed:
$ ls -l /dev/mem /dev/kmem
ls: /dev/kmem: No such file or directory
crw-r----- 1 root kmem 1, 1 Oct 6 09:17 /dev/mem
$
However, the crash utility, if it realizes that it cannot access a physical
address because it's bigger than the high_memory limit, does this in
read_dev_mem():
/*
* /dev/mem disallows anything >= __pa(high_memory)
*
* However it will allow 64-bit lseeks to anywhere, and when followed
* by pulling a 32-bit address from the 64-bit file position, it
* quietly returns faulty data from the (wrapped-around) address.
*/
if (vt->high_memory && (paddr >=
(physaddr_t)(VTOP(vt->high_memory)))) {
readcnt = 0;
errno = 0;
goto try_dev_kmem;
}
So the vmalloc read of 0xf9102280, whose physical address is 0x119b76280 is
greater than the VTOP of 0xf8000000, or b8000000, will goto try_dev_kmem.
First, do you have a /dev/kmem on your system? If so, the read attempt
continues like so if the passed-in address was a vmalloc address:
if ((readcnt != cnt) && BITS32() && !readcnt && !errno
&&
IS_VMALLOC_ADDR(addr))
readcnt = read_dev_kmem(addr, bufptr, cnt);
You'll have to debug crash's memory.c:read_dev_mem() function to determine
whether it followed that path, and successfully read_dev_kmem() above to get
the legitimate module data. That would be a possible (the only?) explanation
for what you're seeing.
Now, that all being said, debugging the above offers nothing towards the
debugging of the kdump/dumpfile issue.
Dave