3:12 p.m.
----- "Kevin Worth" <kevin.worth(a)hp.com> wrote:
Hi Dave,
I tried changing the PAGE_OFFSET definition in kexec-tools. Didn't
seem to affect it- crash still fails to load the vmalloc'ed memory. If
that seems like it absolves kexec-tools of any sins then perhaps we
can drop the kexec-ml off the CC list.
Your statement "Theoretically, anything at and above 0xb8000000 should
fail." was accurate, which I saw on my live system (with no dump
involved). Hoping this provides some insight.
Right -- but when you did the "module 0xf9102280", it read legitimate
data -- but the translated vmalloc address of 0xf9102280 shows a physical
address of 119b76280, i.e. well beyond the physical limit of 0xb8000000:
crash> module 0xf9102280
struct module {
state = MODULE_STATE_LIVE,
list = {
next = 0xf9073d84,
prev = 0x403c63a4
},
name = "custom_lkm"
...
crash> vtop 0xf9102280
VIRTUAL PHYSICAL
f9102280 119b76280
...
crash> rd -p 119b76000 30
rd: read error: physical address: 119b76000 type: "32-bit PHYSADDR"
...
That being the case, I just remembered something that I had completely
forgotten about -- because of yet another Red Hat imposed restriction.
On RHEL systems, we have the restricted /dev/mem, but in addition to
that /dev/kmem has been completely removed:
$ ls -l /dev/mem /dev/kmem
ls: /dev/kmem: No such file or directory
crw-r----- 1 root kmem 1, 1 Oct 6 09:17 /dev/mem
$
However, the crash utility, if it realizes that it cannot access a physical
address because it's bigger than the high_memory limit, does this in
read_dev_mem():
/*
* /dev/mem disallows anything >= __pa(high_memory)
*
* However it will allow 64-bit lseeks to anywhere, and when followed
* by pulling a 32-bit address from the 64-bit file position, it
* quietly returns faulty data from the (wrapped-around) address.
*/
if (vt->high_memory && (paddr >=
(physaddr_t)(VTOP(vt->high_memory)))) {
readcnt = 0;
errno = 0;
goto try_dev_kmem;
}
So the vmalloc read of 0xf9102280, whose physical address is 0x119b76280 is
greater than the VTOP of 0xf8000000, or b8000000, will goto try_dev_kmem.
First, do you have a /dev/kmem on your system? If so, the read attempt
continues like so if the passed-in address was a vmalloc address:
if ((readcnt != cnt) && BITS32() && !readcnt && !errno
&&
IS_VMALLOC_ADDR(addr))
readcnt = read_dev_kmem(addr, bufptr, cnt);
You'll have to debug crash's memory.c:read_dev_mem() function to determine
whether it followed that path, and successfully read_dev_kmem() above to get
the legitimate module data. That would be a possible (the only?) explanation
for what you're seeing.
Now, that all being said, debugging the above offers nothing towards the
debugging of the kdump/dumpfile issue.
Dave
3:25 p.m.
New subject: "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash
Dave,
I do indeed have a /dev/kmem on my live system. One other thing... is the physical limit
of 0xb8000000 imposed because of the amount of memory on my system or because it is the
maximum addressable memory without using PAE? My kernel does have PAE enabled
(CONFIG_HIGHMEM_32G or whatever the option is), though not sure if that makes a
difference, just checking. Things work just fine on an identical system that has 2GB of
memory instead of 4GB. :\
-Kevin
-----Original Message-----
From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf
Of Dave Anderson
Sent: Friday, October 10, 2008 1:13 PM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] "cannot access vmalloc'd module memory" when
loading kdump'ed vmcore in crash
----- "Kevin Worth" <kevin.worth(a)hp.com> wrote:
Hi Dave,
I tried changing the PAGE_OFFSET definition in kexec-tools. Didn't
seem to affect it- crash still fails to load the vmalloc'ed memory. If
that seems like it absolves kexec-tools of any sins then perhaps we
can drop the kexec-ml off the CC list.
Your statement "Theoretically, anything at and above 0xb8000000 should
fail." was accurate, which I saw on my live system (with no dump
involved). Hoping this provides some insight.
Right -- but when you did the "module 0xf9102280", it read legitimate
data -- but the translated vmalloc address of 0xf9102280 shows a physical
address of 119b76280, i.e. well beyond the physical limit of 0xb8000000:
crash> module 0xf9102280
struct module {
state = MODULE_STATE_LIVE,
list = {
next = 0xf9073d84,
prev = 0x403c63a4
},
name = "custom_lkm"
...
crash> vtop 0xf9102280
VIRTUAL PHYSICAL
f9102280 119b76280
...
crash> rd -p 119b76000 30
rd: read error: physical address: 119b76000 type: "32-bit PHYSADDR"
...
That being the case, I just remembered something that I had completely
forgotten about -- because of yet another Red Hat imposed restriction.
On RHEL systems, we have the restricted /dev/mem, but in addition to
that /dev/kmem has been completely removed:
$ ls -l /dev/mem /dev/kmem
ls: /dev/kmem: No such file or directory
crw-r----- 1 root kmem 1, 1 Oct 6 09:17 /dev/mem
$
However, the crash utility, if it realizes that it cannot access a physical
address because it's bigger than the high_memory limit, does this in
read_dev_mem():
/*
* /dev/mem disallows anything >= __pa(high_memory)
*
* However it will allow 64-bit lseeks to anywhere, and when followed
* by pulling a 32-bit address from the 64-bit file position, it
* quietly returns faulty data from the (wrapped-around) address.
*/
if (vt->high_memory && (paddr >=
(physaddr_t)(VTOP(vt->high_memory)))) {
readcnt = 0;
errno = 0;
goto try_dev_kmem;
}
So the vmalloc read of 0xf9102280, whose physical address is 0x119b76280 is
greater than the VTOP of 0xf8000000, or b8000000, will goto try_dev_kmem.
First, do you have a /dev/kmem on your system? If so, the read attempt
continues like so if the passed-in address was a vmalloc address:
if ((readcnt != cnt) && BITS32() && !readcnt && !errno
&&
IS_VMALLOC_ADDR(addr))
readcnt = read_dev_kmem(addr, bufptr, cnt);
You'll have to debug crash's memory.c:read_dev_mem() function to determine
whether it followed that path, and successfully read_dev_kmem() above to get
the legitimate module data. That would be a possible (the only?) explanation
for what you're seeing.
Now, that all being said, debugging the above offers nothing towards the
debugging of the kdump/dumpfile issue.
Dave
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
3:31 p.m.
New subject: "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash
if ((readcnt != cnt) && BITS32() && !readcnt
&& !errno &&
IS_VMALLOC_ADDR(addr))
readcnt = read_dev_kmem(addr, bufptr, cnt);
You'll have to debug crash's memory.c:read_dev_mem() function to determine
whether it followed that path, and successfully read_dev_kmem() above to get
the legitimate module data. That would be a possible (the only?)
explanation for what you're seeing.
I should note that without having to rebuild a debug version of
crash, you can tell whether any /dev/kmem reads have been attempted.
The crash utility's read_dev_kmem() looks like this:
/*
* When /dev/mem won't allow access, try /dev/kmem.
*/
static ssize_t
read_dev_kmem(ulong vaddr, char *bufptr, long cnt)
{
ssize_t readcnt;
if (pc->kfd < 0) {
if ((pc->kfd = open("/dev/kmem", O_RDONLY)) < 0)
return 0;
}
if (lseek(pc->kfd, vaddr, SEEK_SET) == -1)
return 0;
readcnt = read(pc->kfd, bufptr, cnt);
if (readcnt != cnt)
readcnt = 0;
return readcnt;
}
The global file descriptor pc->kfd is initialized to -1, and
so when run on RHEL systems it will remain equal to -1 because
/dev/kmem doesn't exist. Presuming that it does exist on your
system, pc->kfd will get a file descriptor -- and will keep
it open for the remainder of the crash session. You can look
at the value of pc->kfd by entering the debug command "help -p".
Here on on x86 running on RHEL5, pc->kfd is still -1:
crash> help -p
program_name: crash
program_path: crash
program_version: 4.0-7.2
gdb_version: 6.1
program_pid: 7165
prompt: "crash> "
flags: 3200000210c07
(RUNTIME|LIVE_SYSTEM|TTY|HASH|SCROLL|GDB_INIT|READLINE|MEMMOD|DATADEBUG|
FINDKERNEL)
namelist: /usr/lib/debug/lib/modules/2.6.18-118.el5.perfmon2/vmlinux
dumpfile: (null)
live_memsrc: /dev/crash
system_map: (null)
namelist_debug: (null)
debuginfo_file: (null)
memory_module: crash
memory_device: /dev/crash
machine_type: X86
editing_mode: vi
nfd: -1
mfd: 4
kfd: -1
...
If your "help -p" shows a legitimate pc->kfd file descriptor, then
in all probability, that's why you can read module data via its
virtual address -- while at the same time not being able to
access it via its physical address (rd -p).
Dave
4:02 p.m.
New subject: "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash
Yep. Also confirmed using the printf's in previous email.
crash> help -p
program_name: crash
program_path: ./crash
program_version: 4.0-7.2
gdb_version: 6.1
...
nfd: -1
mfd: 4
kfd: 11
-----Original Message-----
From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf
Of Dave Anderson
Sent: Friday, October 10, 2008 1:31 PM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] "cannot access vmalloc'd module memory" when
loading kdump'ed vmcore in crash
if ((readcnt != cnt) && BITS32() && !readcnt
&& !errno &&
IS_VMALLOC_ADDR(addr))
readcnt = read_dev_kmem(addr, bufptr, cnt);
You'll have to debug crash's memory.c:read_dev_mem() function to determine
whether it followed that path, and successfully read_dev_kmem() above to get
the legitimate module data. That would be a possible (the only?)
explanation for what you're seeing.
I should note that without having to rebuild a debug version of
crash, you can tell whether any /dev/kmem reads have been attempted.
The crash utility's read_dev_kmem() looks like this:
/*
* When /dev/mem won't allow access, try /dev/kmem.
*/
static ssize_t
read_dev_kmem(ulong vaddr, char *bufptr, long cnt)
{
ssize_t readcnt;
if (pc->kfd < 0) {
if ((pc->kfd = open("/dev/kmem", O_RDONLY)) < 0)
return 0;
}
if (lseek(pc->kfd, vaddr, SEEK_SET) == -1)
return 0;
readcnt = read(pc->kfd, bufptr, cnt);
if (readcnt != cnt)
readcnt = 0;
return readcnt;
}
The global file descriptor pc->kfd is initialized to -1, and
so when run on RHEL systems it will remain equal to -1 because
/dev/kmem doesn't exist. Presuming that it does exist on your
system, pc->kfd will get a file descriptor -- and will keep
it open for the remainder of the crash session. You can look
at the value of pc->kfd by entering the debug command "help -p".
Here on on x86 running on RHEL5, pc->kfd is still -1:
crash> help -p
program_name: crash
program_path: crash
program_version: 4.0-7.2
gdb_version: 6.1
program_pid: 7165
prompt: "crash> "
flags: 3200000210c07
(RUNTIME|LIVE_SYSTEM|TTY|HASH|SCROLL|GDB_INIT|READLINE|MEMMOD|DATADEBUG|
FINDKERNEL)
namelist: /usr/lib/debug/lib/modules/2.6.18-118.el5.perfmon2/vmlinux
dumpfile: (null)
live_memsrc: /dev/crash
system_map: (null)
namelist_debug: (null)
debuginfo_file: (null)
memory_module: crash
memory_device: /dev/crash
machine_type: X86
editing_mode: vi
nfd: -1
mfd: 4
kfd: -1
...
If your "help -p" shows a legitimate pc->kfd file descriptor, then
in all probability, that's why you can read module data via its
virtual address -- while at the same time not being able to
access it via its physical address (rd -p).
Dave
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
10:27 a.m.
New subject: "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash
Worth, Kevin wrote:
Thanks for the explanation, Dave.
I tried adding some printf's to memory.c (before the goto try_dev_kmem, then
before and after the call to read_dev_kmem). It does in fact look like it's
taking the read_dev_kmem path... a couple examples (a ton of these are printed,
so if there is any specific address to look for let me know, but I presume this
is just confirming what you suspected.
...
Yep. Also confirmed using the printf's in previous email.
crash> help -p
program_name: crash
program_path: ./crash
program_version: 4.0-7.2
gdb_version: 6.1
...
nfd: -1
mfd: 4
kfd: 11
OK, we're getting nowhere fast because of the limitations of
the /dev/mem driver.
What I've been trying to accomplish is simply this:
on the live system:
- find a module address whose "vtop" shows that its physical
address is greater than 4GB.
- rd -p <the-physical-address-greater-than-4GB> 100
- verify that it contains "correct" data, i.e. as seen when
you do a "p <module-virtual-address>" (it will...)
<crash the system>
on the dumpfile:
- rd -p <the-physical-address-greater-than-4GB> 100 (same as above)
- see whether it does -- or does *not* -- contain the "correct" data
that was seen on the live system
If the same physical address does *not* contain the same data
in the dumpfile as was there in the live system, then we can
point at the kexec/kdump operation. We have seemingly done
so, but without being able to do it *exactly* as the steps
above because:
- on the live system, we're relying on /dev/kmem to do the
virtual-to-physical address translation of vmalloc addresses,
so the resultant physical address is "hidden" from us.
If your system used the Red Hat /dev/crash driver, then the
steps above would be trivial, because the crash utility does
does the virtual-to-physical address translation of the
vmalloc addresses itself, and then reads memory using the
resultant physical address.
I will attach the crash.c and crash.h files from /dev/crash patch
we use for RHEL5's 2.6.18-based kernel. What you would need to do
is something like this (having never done it before):
(1) Copy the attached crash.c file to your kernel's "drivers/char"
directory.
(2) Copy the attached crash.h file to your kernel's "include/asm-x86_64"
directory.
(3) Add this to the "drivers/char/Makefile":
+obj-m += crash.o
(4) since 2.6.20 doesn't have a page_is_ram() function for x86_64,
you'll have to add this to your arch/x86_64/mm/init.c file,
and export it so the /dev/crash driver can pick it up:
int page_is_ram (unsigned long pagenr)
{
int i;
for (i = 0; i < e820.nr_map; i++) {
unsigned long addr, end;
if (e820.map[i].type != E820_RAM) /* not usable memory */
continue;
/*
* !!!FIXME!!! Some BIOSen report areas as RAM that
* are not. Notably the 640->1Mb area. We need a sanity
* check here.
*/
addr = (e820.map[i].addr+PAGE_SIZE-1) >> PAGE_SHIFT;
end = (e820.map[i].addr+e820.map[i].size) >> PAGE_SHIFT;
if ((pagenr >= addr) && (pagenr < end))
return 1;
}
return 0;
}
EXPORT_SYMBOL_GPL(page_is_ram);
(5) Build the kernel and see what happens...
Other than trying that, I don't have any other suggestions.
Dave
10:54 a.m.
New subject: "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash
Thanks, Dave. I was actually going to ask about that since you had mentioned that
/dev/crash should behave correctly in contrast to /dev/mem / /dev/kmem. Does it matter
that my arch is i386 and not x86_64? I'll give this a shot.
-Kevin
________________________________________
From: crash-utility-bounces(a)redhat.com [crash-utility-bounces(a)redhat.com] On Behalf Of
Dave Anderson [anderson(a)redhat.com]
Sent: Tuesday, October 14, 2008 8:27 AM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] "cannot access vmalloc'd module memory" when
loading kdump'ed vmcore in crash
Worth, Kevin wrote:
Thanks for the explanation, Dave.
I tried adding some printf's to memory.c (before the goto try_dev_kmem, then
before and after the call to read_dev_kmem). It does in fact look like it's
taking the read_dev_kmem path... a couple examples (a ton of these are printed,
so if there is any specific address to look for let me know, but I presume this
is just confirming what you suspected.
...
Yep. Also confirmed using the printf's in previous email.
crash> help -p
program_name: crash
program_path: ./crash
program_version: 4.0-7.2
gdb_version: 6.1
...
nfd: -1
mfd: 4
kfd: 11
OK, we're getting nowhere fast because of the limitations of
the /dev/mem driver.
What I've been trying to accomplish is simply this:
on the live system:
- find a module address whose "vtop" shows that its physical
address is greater than 4GB.
- rd -p <the-physical-address-greater-than-4GB> 100
- verify that it contains "correct" data, i.e. as seen when
you do a "p <module-virtual-address>" (it will...)
<crash the system>
on the dumpfile:
- rd -p <the-physical-address-greater-than-4GB> 100 (same as above)
- see whether it does -- or does *not* -- contain the "correct" data
that was seen on the live system
If the same physical address does *not* contain the same data
in the dumpfile as was there in the live system, then we can
point at the kexec/kdump operation. We have seemingly done
so, but without being able to do it *exactly* as the steps
above because:
- on the live system, we're relying on /dev/kmem to do the
virtual-to-physical address translation of vmalloc addresses,
so the resultant physical address is "hidden" from us.
If your system used the Red Hat /dev/crash driver, then the
steps above would be trivial, because the crash utility does
does the virtual-to-physical address translation of the
vmalloc addresses itself, and then reads memory using the
resultant physical address.
I will attach the crash.c and crash.h files from /dev/crash patch
we use for RHEL5's 2.6.18-based kernel. What you would need to do
is something like this (having never done it before):
(1) Copy the attached crash.c file to your kernel's "drivers/char"
directory.
(2) Copy the attached crash.h file to your kernel's "include/asm-x86_64"
directory.
(3) Add this to the "drivers/char/Makefile":
+obj-m += crash.o
(4) since 2.6.20 doesn't have a page_is_ram() function for x86_64,
you'll have to add this to your arch/x86_64/mm/init.c file,
and export it so the /dev/crash driver can pick it up:
int page_is_ram (unsigned long pagenr)
{
int i;
for (i = 0; i < e820.nr_map; i++) {
unsigned long addr, end;
if (e820.map[i].type != E820_RAM) /* not usable memory */
continue;
/*
* !!!FIXME!!! Some BIOSen report areas as RAM that
* are not. Notably the 640->1Mb area. We need a sanity
* check here.
*/
addr = (e820.map[i].addr+PAGE_SIZE-1) >> PAGE_SHIFT;
end = (e820.map[i].addr+e820.map[i].size) >> PAGE_SHIFT;
if ((pagenr >= addr) && (pagenr < end))
return 1;
}
return 0;
}
EXPORT_SYMBOL_GPL(page_is_ram);
(5) Build the kernel and see what happens...
Other than trying that, I don't have any other suggestions.
Dave
11:09 a.m.
New subject: "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash
Worth, Kevin wrote:
Thanks, Dave. I was actually going to ask about that since you had
mentioned that
/dev/crash should behave correctly in contrast to /dev/mem / /dev/kmem. Does it
matter that my arch is i386 and not x86_64? I'll give this a shot.
D'oh...
Sorry yes, it sure does -- I've attached the i386 version of crash.h.
(The crash.c file is arch-neutral)
For that matter, 2.6.20 does have an i386 version of page_is_ram(),
so all you'll have to do is add EXPORT_SYMBOL_GPL(page_is_ram) in
"arch/i386/mm/init.c".
Dave
1:29 p.m.
New subject: "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash
Thanks, Dave. Is it valid to just do "make modules" since it appears we're
just adding a module or does the modification to arch/i386/mm/init.c necessitate a rebuilt
kernel?
-Kevin
-----Original Message-----
From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf
Of Dave Anderson
Sent: Tuesday, October 14, 2008 9:09 AM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] "cannot access vmalloc'd module memory" when
loading kdump'ed vmcore in crash
Worth, Kevin wrote:
Thanks, Dave. I was actually going to ask about that since you had
mentioned that > /dev/crash should behave correctly in contrast to /dev/mem /
/dev/kmem. Does it > matter that my arch is i386 and not x86_64? I'll give this a
shot.
D'oh...
Sorry yes, it sure does -- I've attached the i386 version of crash.h.
(The crash.c file is arch-neutral)
For that matter, 2.6.20 does have an i386 version of page_is_ram(), so all you'll have
to do is add EXPORT_SYMBOL_GPL(page_is_ram) in "arch/i386/mm/init.c".
Dave
5887
days inactive
5891
days old
devel@lists.crash-utility.osci.io
7 comments
2 participants
participants (2)
-
Dave Anderson -
Worth, Kevin