4:08 p.m.
I am seeing an issue when opening a crash file from RHEL AS 2.1 with the last 2.1 kernel
released where crash will sit at "please wait... (gathering kmem slab cache
data)" forever taking 100% of the CPU. Turning on debugging at level 5 continuously
prints out the following sequence over and over without stopping:
<readmem: f79d52cc, KVADDR, "kmem_cache buffer", 244, (ROE), 8585500>
<readmem: f79d5340, KVADDR, "cpudata array", 128, (ROE), ffffcbd0>
<readmem: f79d5344, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5344, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d534c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d534c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5354, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5354, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d535c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d535c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5364, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5364, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d536c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d536c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5374, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5374, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d537c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d537c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5384, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5384, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d538c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d538c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5394, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5394, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d539c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d539c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d53a4, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d53a4, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d53ac, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d53ac, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d53b4, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d53b4, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d53bc, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d53bc, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d52cc, KVADDR, "kmem_cache buffer", 244, (ROE), 8585500>
....
I am on the latest version of crash, 5.1.3:
[srhode@crash 1642489]$ crash32 -v
crash32 5.1.3
Copyright (C) 2002-2011 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu
--target=i686-pc-linux-gnu".
If I open the dump file using the "-no_kmem_cache" option, it opens, but I am
not sure if this is causing other issues during analysis of the dump file.
Shawn Rhode
Principal Support Engineer - Egenera Global Technical Support
8:40 a.m.
----- Original Message -----
I am seeing an issue when opening a crash file from RHEL AS 2.1 with
the last 2.1 kernel released where crash will sit at “please wait...
(gathering kmem slab cache data)” forever taking 100% of the CPU.
Turning on debugging at level 5 continuously prints out the following
sequence over and over without stopping:
<readmem: f79d52cc, KVADDR, "kmem_cache buffer", 244, (ROE), 8585500>
<readmem: f79d5340, KVADDR, "cpudata array", 128, (ROE), ffffcbd0>
<readmem: f79d5344, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5344, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d534c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d534c, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d5354, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
... [ cut ] ...
<readmem: f79d53a4, KVADDR, "cpucache limit", 4, (ROE),
ffffcbcc>
<readmem: f79d53ac, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d53ac, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d53b4, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d53b4, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d53bc, KVADDR, "cpucache limit", 4, (ROE), ffffcbcc>
<readmem: f79d52cc, KVADDR, "kmem_cache buffer", 244, (ROE), 8585500>
....
If you run crash on the live system running that kernel, or on any
other dumps running that kernel version, does crash initialize?
(I'm sorry -- that kernel is so old I don't even have any sample
dumpfiles left hanging around...) Somehow kmem_cache_init() is
looping back on itself when walking through the linked-list of kmem_cache
structures. But that's such a fundamental flaw, that I'd really like
to know if somehow over the years something has changed such that the
mechanism used for following the linked-list no longer works for a kernel
of that vintage?
Anyway, there's no way of working around that other than what you have
done by using --no_kmem_cache. That doesn't cause any other functional
problems other than to set an internal KMEM_CACHE_UNAVAIL flag that
is used to disallow command options that need to access the kmem slab
substructure.
Other than that, there's nothing else that can be done unless you
want to start tinkering with the crash code. It would be a matter
of determining why the linked list cannot be followed. If you want
to pursue that, I'd suggest playing around with the kmem_cache_list()
function in memory.c. You can bring the crash session up with
--no_kmem_cache, and then run the "kmem -s list" function, which
calls kmem_cache_list(). That function just walks the kmem_cache
linked list dumping out the name and address of each slab cache
in the same way the kmem_cache_init() does -- but you would first
have to comment out the check for the KMEM_CACHE_UNAVAIL flag at
the top of the function:
static void
kmem_cache_list(void)
{
ulong cache, cache_cache, name;
long next_offset, name_offset;
char *cache_buf;
int has_cache_chain;
ulong cache_chain;
char buf[BUFSIZE];
if (vt->flags & KMEM_CACHE_UNAVAIL) {
error(INFO, "kmem cache slab subsystem not available\n");
return;
}
Anyway, doing that, you can see how/where the linked-list goes back
on itself.
But it's really important to first determine whether this is always
going to fail with that kernel, or if it's specific to that particular
dumpfile.
Dave
5004
days inactive
5005
days old
devel@lists.crash-utility.osci.io
1 comments
2 participants
participants (2)
-
Dave Anderson -
Shawn Rhode