6:58 a.m.
Hi Dave,
I have a corrupt vmcore file (for ARM) that makes crash loop forever. The problem is in
memory.c, function max_cpudata_limit. The last part of that function:
if (VALID_MEMBER(kmem_list3_shared) && VALID_MEMBER(kmem_cache_s_lists)
&&
readmem(kmem_cache_nodelists(cache), KVADDR, &start_address[0],
sizeof(ulong) * vt->kmem_cache_len_nodes, "array nodelist
array",
RETURN_ON_ERROR)) {
for (i = 0; i < vt->kmem_cache_len_nodes; i++) {
if (start_address[i] == 0)
continue;
if (readmem(start_address[i] + OFFSET(kmem_list3_shared),
KVADDR, &shared, sizeof(void *),
"kmem_list3 shared", RETURN_ON_ERROR|QUIET)) {
if (!shared)
break;
}
if (readmem(shared + OFFSET(array_cache_limit),
KVADDR, &limit, sizeof(int), "shared array_cache
limit",
RETURN_ON_ERROR|QUIET)) {
if (limit > max_limit)
max_limit = limit;
break;
}
}
}
FREEBUF(start_address);
return max_limit;
bail_out:
vt->flags |= KMEM_CACHE_UNAVAIL;
error(INFO, "unable to initialize kmem slab cache subsystem\n\n");
*cpus = 0;
return 0;
The problem is that the readmem statement "if (readmem(start_address[i] +
OFFSET(kmem_list3_shared), ....." fails, and then the function max_cpudata_limit is
called over and over again. I did a patch adding "else goto bail_out;" if the
readmem fails and then crash managed to continue. I do not know if this is really a good
idea.
As this seems only to be a problem for corrupt vmcore files I do not know if you want to
do anything about it.
Best regards
Jan
Jan Karlsson
Senior Software Engineer
MIB
Sony Mobile Communications
Tel: +46703062174
sonymobile.com<http://sonymobile.com/>
[cid:image001.jpg@01CD22E9.EA9DB1E0]
9:42 a.m.
----- Original Message -----
Hi Dave,
I have a corrupt vmcore file (for ARM) that makes crash loop forever.
The problem is in memory.c, function max_cpudata_limit. The last
part of that function:
if (VALID_MEMBER(kmem_list3_shared) &&
VALID_MEMBER(kmem_cache_s_lists) &&
readmem(kmem_cache_nodelists(cache), KVADDR, &start_address[0],
sizeof(ulong) * vt->kmem_cache_len_nodes, "array nodelist array",
RETURN_ON_ERROR)) {
for (i = 0; i < vt->kmem_cache_len_nodes; i++) {
if (start_address[i] == 0)
continue;
if (readmem(start_address[i] + OFFSET(kmem_list3_shared),
KVADDR, &shared, sizeof(void *),
"kmem_list3 shared", RETURN_ON_ERROR|QUIET)) {
if (!shared)
break;
}
if (readmem(shared + OFFSET(array_cache_limit),
KVADDR, &limit, sizeof(int), "shared array_cache limit",
RETURN_ON_ERROR|QUIET)) {
if (limit > max_limit)
max_limit = limit;
break;
}
}
}
FREEBUF(start_address);
return max_limit;
bail_out:
vt->flags |= KMEM_CACHE_UNAVAIL;
error(INFO, "unable to initialize kmem slab cache subsystem\n\n");
*cpus = 0;
return 0;
The problem is that the readmem statement “if
(readmem(start_address[i] + OFFSET(kmem_list3_shared), …..” fails,
and then the function max_cpudata_limit is called over and over
again. I did a patch adding “else goto bail_out;” if the readmem
fails and then crash managed to continue. I do not know if this is
really a good idea.
As this seems only to be a problem for corrupt vmcore files I do not
know if you want to do anything about it.
Maybe -- maybe not...
In the case of corrupted vmcores, it's preferable to avoid a cover-up,
and in fact, the crash utility is often "doing its job" by failing,
i,e., its failure points to the problem at hand.
However, in the specific case of the kmem_cache initialization, that has
been a problem area in the past when the subsystem itself is corrupted,
or perhaps in your case where the vmcore is corrupted. That's why
the "crash --no_kmem_cache" or "crash --kmem_cache_delay" options
were put in place.
Now in your case, I'm guessing that the crash session may have
quietly "hung" during initialization? And with debug turned on you
may have seen the readmem failures?
I tried to reproduce this by injecting a readmem() failure for
that particular readmem(), but it does not result in a loop.
In my test, the readmem() fails, max_cpudata_limit() eventually returns,
and kmem_cache_init() just goes onto the next kmem_cache in the chain.
Also, because that readmem() is explicitly set RETURN_ON_ERROR|QUIET, it can
conceivably fail without max_cpudata_limit() having to set KMEM_CACHE_UNAVAIL.
Anyway, if max_cpudata_limit() returns without setting KMEM_CACHE_UNAVAIL,
kmem_cache_init() should just continue to walk through the kmem_cache
chain:
[ initialize "cache" and "cache_end" ]
do {
... [ cut ] ...
if ((tmp = max_cpudata_limit(cache, &tmp2)) > max_limit)
max_limit = tmp;
/*
* Recognize and bail out on any max_cpudata_limit() failures.
*/
if (vt->flags & KMEM_CACHE_UNAVAIL) {
FREEBUF(cache_buf);
return;
}
... [ cut ] ...
cache = ULONG(cache_buf + next_offset);
switch (vt->flags & (PERCPU_KMALLOC_V1|PERCPU_KMALLOC_V2))
{
case PERCPU_KMALLOC_V1:
cache -= next_offset;
break;
case PERCPU_KMALLOC_V2:
if (cache != cache_end)
cache -= next_offset;
break;
}
} while (cache != cache_end)
So I don't understand how you got into a loop unless the kmem_cache list
walk-through is the real problem. If you were to print out the "cache"
address each time through the do-while loop, does the list start repeating
itself?
And if that's true, perhaps the kmem_cache_init() should use the
hq_open()/hq_enter()/hq_close() facility on each cache address to
catch a duplicate (false) entry.
Dave
9:57 a.m.
----- Original Message -----
So I don't understand how you got into a loop unless the kmem_cache list
walk-through is the real problem. If you were to print out the "cache"
address each time through the do-while loop, does the list start repeating
itself?
And if that's true, perhaps the kmem_cache_init() should use the
hq_open()/hq_enter()/hq_close() facility on each cache address to
catch a duplicate (false) entry.
Dave
As a side issue, you have pinpointed a potential problem
area if the first readmem() does fail, because in that case it
should "continue" instead of using the invalid "shared" value
in the second readmem():
if (readmem(start_address[i] + OFFSET(kmem_list3_shared),
KVADDR, &shared, sizeof(void *),
"kmem_list3 shared", RETURN_ON_ERROR|QUIET)) {
if (!shared)
break;
}
if (readmem(shared + OFFSET(array_cache_limit),
KVADDR, &limit, sizeof(int), "shared array_cache
limit",
RETURN_ON_ERROR|QUIET)) {
if (limit > max_limit)
max_limit = limit;
break;
}
But again, I don't see that having anything to do with your problem.
And in all practical circumstances, that first readmem() should
never fail, even though it is allowable.
I'll fix that...
Dave
1:29 p.m.
----- Original Message -----
So I don't understand how you got into a loop unless the kmem_cache list
walk-through is the real problem. If you were to print out the "cache"
address each time through the do-while loop, does the list start repeating
itself?
And if that's true, perhaps the kmem_cache_init() should use the
hq_open()/hq_enter()/hq_close() facility on each cache address to
catch a duplicate (false) entry.
And if that's true, does the attached patch help?
Thanks,
Dave
1:06 a.m.
Hi
and thanks for your work with this problem.
As you expected crash silently just loops and I spotted the problem by turning on debug
printouts.
If I include printouts for the "cache" address, the first value seems
reasonable, but then it starts to repeat with the value 0x00000001.
Last, your patch solves the problem nicely. I get a warning about duplicate kmem_slab
entry and crash continues to execute and issues other warnings indicating a corrupt vmcore
file.
Jan
Jan Karlsson
Senior Software Engineer
MIB
Sony Mobile Communications
Tel: +46703062174
sonymobile.com
-----Original Message-----
From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf
Of Dave Anderson
Sent: onsdag den 25 april 2012 20:30
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] loop in crash
----- Original Message -----
So I don't understand how you got into a loop unless the kmem_cache
list walk-through is the real problem. If you were to print out the "cache"
address each time through the do-while loop, does the list start
repeating itself?
And if that's true, perhaps the kmem_cache_init() should use the
hq_open()/hq_enter()/hq_close() facility on each cache address to
catch a duplicate (false) entry.
And if that's true, does the attached patch help?
Thanks,
Dave
8:08 a.m.
----- Original Message -----
Hi
and thanks for your work with this problem.
As you expected crash silently just loops and I spotted the problem
by turning on debug printouts.
If I include printouts for the "cache" address, the first value seems
reasonable, but then it starts to repeat with the value 0x00000001.
Last, your patch solves the problem nicely. I get a warning about
duplicate kmem_slab entry and crash continues to execute and issues
other warnings indicating a corrupt vmcore file.
Jan
OK good -- I should have hq_xxx()'d that loop a long time ago.
Queued for crash-6.0.6.
Thanks,
Dave
2:15 a.m.
Thanks Dave.
I found one more issue with a somewhat "corrupt" vmcore. In this case it is
ARM-specific in unwind_arm.c, so maybe Mika will also look at it.
In the case I am investigating I get a readmem error while reading the unwind tables. The
way unwinding currently is implemented Crash then stops and no further analysis is
possible. When I patched Crash to continue anyhow, every command I tried worked nicely
including bt, so there is no reason to stop at this kind of problem.
When investigating further I found that the problem occurs in init_module_unwind_tables.
It is in the call to do_list(&ld) that the readmem error is found. I also looked in
the code for do_list and saw that it could be configured to return even if errors were
found, by setting ld.flags.
/*
* Iterate through unwind table list and store start address of each
* table in table_list.
*/
ld.flags += RETURN_ON_LIST_ERROR; /* added line */
hq_open();
cnt = do_list(&ld);
if (cnt == -1) { /* added if statement, 3 lines */
return FALSE;
}
table_list = (ulong *)GETBUF(cnt * sizeof(ulong));
cnt = retrieve_list(table_list, cnt);
hq_close();
By adding the lines indicated above I get an appropriate warning that the unwind tables
cannot be read, and then Crash works as usual.
Jan
Jan Karlsson
Senior Software Engineer
MIB
Sony Mobile Communications
Tel: +46703062174
sonymobile.com
-----Original Message-----
From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf
Of Dave Anderson
Sent: torsdag den 26 april 2012 15:09
To: Discussion list for crash utility usage, maintenance and development
Cc: Fänge, Thomas
Subject: Re: [Crash-utility] loop in crash
----- Original Message -----
Hi
and thanks for your work with this problem.
As you expected crash silently just loops and I spotted the problem
by turning on debug printouts.
If I include printouts for the "cache" address, the first value seems
reasonable, but then it starts to repeat with the value 0x00000001.
Last, your patch solves the problem nicely. I get a warning about
duplicate kmem_slab entry and crash continues to execute and issues
other warnings indicating a corrupt vmcore file.
Jan
OK good -- I should have hq_xxx()'d that loop a long time ago.
Queued for crash-6.0.6.
Thanks,
Dave
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
8:28 a.m.
----- Original Message -----
Thanks Dave.
I found one more issue with a somewhat "corrupt" vmcore. In this case
it is ARM-specific in unwind_arm.c, so maybe Mika will also look at
it.
In the case I am investigating I get a readmem error while reading
the unwind tables. The way unwinding currently is implemented Crash
then stops and no further analysis is possible. When I patched Crash
to continue anyhow, every command I tried worked nicely including
bt, so there is no reason to stop at this kind of problem.
When investigating further I found that the problem occurs in
init_module_unwind_tables. It is in the call to do_list(&ld) that
the readmem error is found. I also looked in the code for do_list
and saw that it could be configured to return even if errors were
found, by setting ld.flags.
/*
* Iterate through unwind table list and store start address of each
* table in table_list.
*/
ld.flags += RETURN_ON_LIST_ERROR; /* added line */
hq_open();
cnt = do_list(&ld);
if (cnt == -1) { /* added if statement, 3 lines */
return FALSE;
}
table_list = (ulong *)GETBUF(cnt * sizeof(ulong));
cnt = retrieve_list(table_list, cnt);
hq_close();
By adding the lines indicated above I get an appropriate warning that
the unwind tables cannot be read, and then Crash works as usual.
Jan
Your patch makes perfect sense. Any error(FATAL, ...) call prior to
RUNTIME being set kills the whole session. But if it is possible for
the session to continue, then it should be allowed to.
I'll also add an unwind-specific warning message, and make the same
change to the x86_64 populate_local_tables() function, upon which it
appears that the ARM version was based.
Queued for crash-6.0.6. (Later today...)
Thanks,
Dave
4596
days inactive
4598
days old
devel@lists.crash-utility.osci.io
7 comments
2 participants
participants (2)
-
Dave Anderson -
Karlsson, Jan