[PATCH v2 0/3] Add support for kernel aslr
by Andy Honig
Changes from v1:
Separated changes in x86_64 to a separate file as they are only related
in that I needed to add support for slighty different kernels in order
to fully test these patches.
Corrected an alignment check from "& 0x1000 == 0" to "&0xFFF == 0"
Andy Honig (3):
Improve search for __switch_to symbol
Add support to kaslr for offline vmcore files.
Automatically detect kernel aslr offset.
defs.h | 2 ++
main.c | 8 +++--
netdump.c | 11 ++++++
symbols.c | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++------------
x86_64.c | 20 +++++++----
5 files changed, 127 insertions(+), 31 deletions(-)
--
1.9.0.rc1.175.g0b1dcb5
10 years, 6 months
[PATCH] Extensions: swap_usage: Add thread group leader support
by Aaron Tomlin
Provide the -G option to display the swap consumption
of the thread group leader in a thread group.
For example:
crash> pswap -k -G | head
PID SWAP COMM
1 136k systemd
469 284k zsh
599 148k systemd-journal
608 1192k lvmetad
637 1712k systemd-udevd
822 308k auditd
836 104k audispd
838 156k sedispatch
842 92k alsactl
The complete file can be found here:
http://people.redhat.com/~atomlin/crash/extensions/swap_usage.c
Signed-off-by: Aaron Tomlin <atomlin(a)redhat.com>
---
extensions/swap_usage.c | 70 ++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 60 insertions(+), 10 deletions(-)
diff --git a/extensions/swap_usage.c b/extensions/swap_usage.c
index 4f92e8c..d9467ca 100644
--- a/extensions/swap_usage.c
+++ b/extensions/swap_usage.c
@@ -19,6 +19,7 @@
#include "defs.h"
#define DISPLAY_KB (0x2)
+#define DISPLAY_TG (0x4)
#ifdef ARM
#define _PAGE_FILE (1 << 2)
@@ -138,8 +139,8 @@ show_swap_usage(struct task_context *tc, ulong exists, ulong flag)
if (flag & DISPLAY_KB)
swap_usage <<= (PAGESHIFT()-10);
- fprintf(fp, "%5ld %5ld%s%5s\n",
- tc->pid, swap_usage, (flag & DISPLAY_KB) ? "k\t" : "\t", tc->comm);
+ fprintf(fp, "%3ld %6ld%s%5s\n", tc->pid, swap_usage,
+ (flag & DISPLAY_KB) ? "k\t" : "\t", tc->comm);
}
@@ -151,6 +152,7 @@ cmd_pswap(void)
int c;
ulong value;
ulong flag = 0;
+ ulong tgid;
int subsequent = 0;
ulong exists = MEMBER_NOT_FOUND;
@@ -159,11 +161,14 @@ cmd_pswap(void)
exists = MEMBER_FOUND;
}
- while ((c = getopt(argcnt, args, "k")) != EOF) {
+ while ((c = getopt(argcnt, args, "kG")) != EOF) {
switch (c) {
case 'k':
flag |= DISPLAY_KB;
break;
+ case 'G':
+ flag |= DISPLAY_TG;
+ break;
default:
argerrs++;
break;
@@ -177,9 +182,16 @@ cmd_pswap(void)
PRINT_HEADER();
tc = FIRST_CONTEXT();
for (i = 0; i < RUNNING_TASKS(); i++, tc++) {
- if (!is_kernel_thread(tc->task))
- show_swap_usage(tc, exists, flag);
- }
+ if (!is_kernel_thread(tc->task)) {
+ if (flag & DISPLAY_TG) {
+ tgid = task_tgid(tc->task);
+ if (tc->pid != tgid)
+ continue;
+ tc = tgid_to_context(tgid);
+ }
+ show_swap_usage(tc, exists, flag);
+ }
+ }
return;
}
@@ -189,6 +201,12 @@ cmd_pswap(void)
case STR_PID:
for (tc = pid_to_context(value); tc; tc = tc->tc_next) {
if (!is_kernel_thread(tc->task)) {
+ if (flag & DISPLAY_TG) {
+ tgid = task_tgid(tc->task);
+ if (tc->pid != tgid)
+ continue;
+ tc = tgid_to_context(tgid);
+ }
show_swap_usage(tc, exists, flag);
} else {
error(INFO, "only specify a user task or pid: %s\n",
@@ -200,6 +218,12 @@ cmd_pswap(void)
case STR_TASK:
for (; tc; tc = tc->tc_next) {
if (!is_kernel_thread(tc->task)) {
+ if (flag & DISPLAY_TG) {
+ tgid = task_tgid(tc->task);
+ if (tc->pid != tgid)
+ continue;
+ tc = tgid_to_context(tgid);
+ }
show_swap_usage(tc, exists, flag);
} else {
error(INFO, "only specify a user task or pid: %s\n",
@@ -222,19 +246,45 @@ cmd_pswap(void)
char *help_pswap[] = {
"pswap",
"Returns the actual swap consumption of a user process",
- "[-k] [pid | taskp]",
+ "[-k -G] [pid | taskp]",
" This command obtains the swap consumption (in pages) of a user process.",
- " The -k option can be used to print in kilobytes.\n"
+ " The process list may be restricted with the following options:\n",
+ " -k print in kilobytes.\n"
+ " -G show only the thread group leader in a thread group.\n"
+ " ",
" If no arguments are specified, every user process will be checked.",
" Supported on ARM, X86, X86_64, ALPHA, IA64 and S390 only.",
"\nEXAMPLE",
" Show the swap consumption for pid 1232, 1353 and 2275:\n",
" crash> pswap 1232 1353 2275",
- " PID SWAP COMM",
+ " PID SWAP COMM",
" 1232 34 auditd",
" 1353 526 vi",
" 2275 30237 gnome-shell",
- " crash>",
+ " crash>",
+ " ",
+ " Show the swap consumption for thread group leaders only:\n",
+ " crash> pswap -G",
+ " PID SWAP COMM",
+ " 469 71 zsh",
+ " 599 37 systemd-journal",
+ " 608 298 lvmetad",
+ " 637 428 systemd-udevd",
+ " 822 77 auditd",
+ " 836 26 audispd",
+ " 838 39 sedispatch",
+ " 842 23 alsactl",
+ " 844 44 bluetoothd",
+ " 851 46 rtkit-daemon",
+ " 852 59 accounts-daemon",
+ " 855 23 avahi-daemon",
+ " 857 96 rsyslogd",
+ " 858 179 restorecond",
+ " 859 144 smartd",
+ " 862 33 irqbalance",
+ " 867 41 systemd-logind",
+ " 868 37 dbus-daemon",
+ " crash>",
NULL
};
--
1.8.5.3
10 years, 6 months
[PATCH 1/2] Add support to kaslr offset for offline vmcore files.
by Andy Honig
[PATCH 1/2] Add support to kaslr for offline vmcore files.
This patch adds a --kaslr command line parameter for loading x86_64
crash dumps with kaslr enabled. This reuses the code from 32-bit
x86 relocations with some small changes. The ASLR offset is postive
instead of negative. Also had to move the code to traverse the
kernel section before the symbol storing code to figure out which
symbols were outside any sections and therefore were not relocated.
Also made a very small change in search_for_switch_to it was
searching through gdb command output for a slightly incorrect syntax.
Tested: Tested by loading kdump files from kernels with aslr enabled
and not enabled. Ran bt, files, and struct file 0xXXXXXX.
Signed-off-by: Andy Honig <ahonig(a)google.com>
---
defs.h | 2 ++
main.c | 8 ++++++--
symbols.c | 66 +++++++++++++++++++++++++++++++++++++++++++++------------------
x86_64.c | 20 +++++++++++++------
4 files changed, 69 insertions(+), 27 deletions(-)
diff --git a/defs.h b/defs.h
index 4cae5e5..c9a4b73 100644
--- a/defs.h
+++ b/defs.h
@@ -2417,6 +2417,8 @@ struct symbol_table_data {
ulong __per_cpu_end;
off_t dwarf_debug_frame_file_offset;
ulong dwarf_debug_frame_size;
+ ulong first_section_start;
+ ulong last_section_end;
};
/* flags for st */
diff --git a/main.c b/main.c
index e5829bc..39763da 100644
--- a/main.c
+++ b/main.c
@@ -57,6 +57,7 @@ static struct option long_options[] = {
{"CRASHPAGER", 0, 0, 0},
{"no_scroll", 0, 0, 0},
{"reloc", required_argument, 0, 0},
+ {"kaslr", required_argument, 0, 0},
{"active", 0, 0, 0},
{"minimal", 0, 0, 0},
{"mod", required_argument, 0, 0},
@@ -216,12 +217,15 @@ main(int argc, char **argv)
else if (STREQ(long_options[option_index].name, "mod"))
kt->module_tree = optarg;
- else if (STREQ(long_options[option_index].name, "reloc")) {
+ else if (STREQ(long_options[option_index].name, "reloc") ||
+ STREQ(long_options[option_index].name, "kaslr")) {
if (!calculate(optarg, &kt->relocate, NULL, 0)) {
error(INFO, "invalid --reloc argument: %s\n",
optarg);
program_usage(SHORT_FORM);
- }
+ } else if (STREQ(long_options[option_index].name, "kaslr")) {
+ kt->relocate *= -1;
+ }
kt->flags |= RELOC_SET;
}
diff --git a/symbols.c b/symbols.c
index d063a0a..28378ab 100644
--- a/symbols.c
+++ b/symbols.c
@@ -198,22 +198,6 @@ symtab_init(void)
no_debugging_data(FATAL);
}
- symcount = bfd_read_minisymbols(st->bfd, FALSE, &minisyms, &size);
-
- if (symcount <= 0)
- no_debugging_data(FATAL);
-
- sort_x = bfd_make_empty_symbol(st->bfd);
- sort_y = bfd_make_empty_symbol(st->bfd);
- if (sort_x == NULL || sort_y == NULL)
- error(FATAL, "bfd_make_empty_symbol() failed\n");
-
- gnu_qsort(st->bfd, minisyms, symcount, size, sort_x, sort_y);
-
- store_symbols(st->bfd, FALSE, minisyms, symcount, size);
-
- free(minisyms);
-
/*
* Gather references to the kernel sections.
*/
@@ -222,6 +206,7 @@ symtab_init(void)
error(FATAL, "symbol table section array malloc: %s\n",
strerror(errno));
BZERO(st->sections, st->bfd->section_count * sizeof(struct sec *));
+ st->first_section_start = st->last_section_end = 0;
bfd_map_over_sections(st->bfd, section_header_info, KERNEL_SECTIONS);
if ((st->flags & (NO_SEC_LOAD|NO_SEC_CONTENTS)) ==
@@ -233,6 +218,22 @@ symtab_init(void)
}
}
+ symcount = bfd_read_minisymbols(st->bfd, FALSE, &minisyms, &size);
+
+ if (symcount <= 0)
+ no_debugging_data(FATAL);
+
+ sort_x = bfd_make_empty_symbol(st->bfd);
+ sort_y = bfd_make_empty_symbol(st->bfd);
+ if (sort_x == NULL || sort_y == NULL)
+ error(FATAL, "bfd_make_empty_symbol() failed\n");
+
+ gnu_qsort(st->bfd, minisyms, symcount, size, sort_x, sort_y);
+
+ store_symbols(st->bfd, FALSE, minisyms, symcount, size);
+
+ free(minisyms);
+
symname_hash_init();
symval_hash_init();
}
@@ -590,7 +591,7 @@ store_symbols(bfd *abfd, int dynamic, void *minisyms, long symcount,
st->symcnt = 0;
sp = st->symtable;
- if (machine_type("X86")) {
+ if (machine_type("X86") || machine_type("X86_64")) {
if (!(kt->flags & RELOC_SET))
kt->flags |= RELOC_FORCE;
} else
@@ -663,7 +664,7 @@ store_sysmap_symbols(void)
error(FATAL, "symbol table namespace malloc: %s\n",
strerror(errno));
- if (!machine_type("X86"))
+ if (!machine_type("X86") && !machine_type("X86_64"))
kt->flags &= ~RELOC_SET;
first = 0;
@@ -735,7 +736,20 @@ relocate(ulong symval, char *symname, int first_symbol)
break;
}
- return (symval - kt->relocate);
+ if (machine_type("X86_64")) {
+ /*
+ * There are some symbols which are outside of any section
+ * either because they are offsets or because they are absolute
+ * addresses. These should not be relocated.
+ */
+ if (symval >= st->first_section_start &&
+ symval <= st->last_section_end) {
+ return symval - kt->relocate;
+ } else {
+ return symval;
+ }
+ } else
+ return symval - kt->relocate;
}
/*
@@ -9679,6 +9693,7 @@ section_header_info(bfd *bfd, asection *section, void *reqptr)
struct load_module *lm;
ulong request;
asection **sec;
+ ulong section_end_address;
request = ((ulong)reqptr);
@@ -9697,6 +9712,11 @@ section_header_info(bfd *bfd, asection *section, void *reqptr)
kt->etext_init = kt->stext_init +
(ulong)bfd_section_size(bfd, section);
}
+
+ if (STREQ(bfd_get_section_name(bfd, section), ".text")) {
+ st->first_section_start = (ulong)
+ bfd_get_section_vma(bfd, section);
+ }
if (STREQ(bfd_get_section_name(bfd, section), ".text") ||
STREQ(bfd_get_section_name(bfd, section), ".data")) {
if (!(bfd_get_section_flags(bfd, section) & SEC_LOAD))
@@ -9713,6 +9733,14 @@ section_header_info(bfd *bfd, asection *section, void *reqptr)
st->dwarf_debug_frame_file_offset = (off_t)section->filepos;
st->dwarf_debug_frame_size = (ulong)bfd_section_size(bfd, section);
}
+
+ if (st->first_section_start != 0) {
+ section_end_address =
+ (ulong) bfd_get_section_vma(bfd, section) +
+ (ulong) bfd_section_size(bfd, section);
+ if (section_end_address > st->last_section_end)
+ st->last_section_end = section_end_address;
+ }
break;
case (ulong)MODULE_SECTIONS:
diff --git a/x86_64.c b/x86_64.c
index 8508e4f..b51e285 100644
--- a/x86_64.c
+++ b/x86_64.c
@@ -5414,16 +5414,22 @@ search_for_switch_to(ulong start, ulong end)
{
ulong max_instructions, address;
char buf1[BUFSIZE];
- char buf2[BUFSIZE];
+ char search_string1[BUFSIZE];
+ char search_string2[BUFSIZE];
int found;
max_instructions = end - start;
found = FALSE;
sprintf(buf1, "x/%ldi 0x%lx", max_instructions, start);
- if (symbol_exists("__switch_to"))
- sprintf(buf2, "callq 0x%lx", symbol_value("__switch_to"));
- else
- buf2[0] = NULLCHAR;
+ if (symbol_exists("__switch_to")) {
+ sprintf(search_string1,
+ "callq 0x%lx", symbol_value("__switch_to"));
+ sprintf(search_string2,
+ "call 0x%lx", symbol_value("__switch_to"));
+ } else {
+ search_string1[0] = NULLCHAR;
+ search_string2[0] = NULLCHAR;
+ }
open_tmpfile();
@@ -5436,7 +5442,9 @@ search_for_switch_to(ulong start, ulong end)
break;
if (strstr(buf1, "<__switch_to>"))
found = TRUE;
- if (strlen(buf2) && strstr(buf1, buf2))
+ if (strlen(search_string1) && strstr(buf1, search_string1))
+ found = TRUE;
+ if (strlen(search_string2) && strstr(buf1, search_string2))
found = TRUE;
}
close_tmpfile();
--
1.9.0.rc1.175.g0b1dcb5
10 years, 6 months
[PATCH 2/2] Automatically detect kernel aslr offset.
by Andy Honig
[PATCH 2/2] Automatically detect kernel aslr offset.
This patch improves support for kernel aslr by automatically
finding the aslr offset based on the _stext symbol in the vmcore
info.
Signed-off-by: Andrew Honig <ahonig(a)google.com>
---
netdump.c | 11 +++++++++++
symbols.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++----
2 files changed, 58 insertions(+), 4 deletions(-)
diff --git a/netdump.c b/netdump.c
index 7dc2fca..884dd73 100644
--- a/netdump.c
+++ b/netdump.c
@@ -411,6 +411,17 @@ is_netdump(char *file, ulong source_query)
get_log_from_vmcoreinfo(file, vmcoreinfo_read_string);
}
+ /*
+ * We may need the _stext_SYMBOL from the vmcore_info to adjust for
+ * kaslr and we may not have gotten it elsewhere.
+ */
+ if (source_query == KDUMP_LOCAL) {
+ char *tmpstring = vmcoreinfo_read_string("SYMBOL(_stext)");
+ kt->vmcoreinfo._stext_SYMBOL =
+ htol(tmpstring, RETURN_ON_ERROR, NULL);
+ free(tmpstring);
+ }
+
return nd->header_size;
bailout:
diff --git a/symbols.c b/symbols.c
index 28378ab..a2fc150 100644
--- a/symbols.c
+++ b/symbols.c
@@ -557,6 +557,43 @@ strip_symbol_end(const char *name, char *buf)
}
/*
+ * Derives the kernel aslr offset by comparing the _stext symbol from the
+ * the vmcore_info in the dump file to the _stext symbol in the vmlinux file.
+ */
+static void
+derive_kaslr_offset(bfd *abfd, int dynamic, bfd_byte *start, bfd_byte *end,
+ unsigned int size, asymbol *store)
+{
+ symbol_info syminfo;
+ asymbol *sym;
+ char *name;
+ unsigned long relocate;
+ char buf[BUFSIZE];
+
+ for (; start < end; start += size) {
+ sym = bfd_minisymbol_to_symbol(abfd, dynamic, start, store);
+ if (sym == NULL)
+ error(FATAL, "bfd_minisymbol_to_symbol() failed\n");
+
+ bfd_get_symbol_info(abfd, sym, &syminfo);
+ name = strip_symbol_end(syminfo.name, buf);
+ if (strcmp("_stext", name) == 0) {
+ relocate = syminfo.value - kt->vmcoreinfo._stext_SYMBOL;
+ /*
+ *To avoid mistaking an mismatched kernel version with
+ * a kaslr offset, we make sure that the offset is
+ * aligned by 0x1000, as it always will be for
+ * kaslr.
+ */
+ if ((relocate & 0xFFF) == 0) {
+ kt->relocate = relocate;
+ kt->flags |= RELOC_SET;
+ }
+ }
+ }
+}
+
+/*
* Store the symbols gathered by symtab_init(). The symbols are stored
* in increasing numerical order.
*/
@@ -591,15 +628,21 @@ store_symbols(bfd *abfd, int dynamic, void *minisyms, long symcount,
st->symcnt = 0;
sp = st->symtable;
+ first = 0;
+ from = (bfd_byte *) minisyms;
+ fromend = from + symcount * size;
+
if (machine_type("X86") || machine_type("X86_64")) {
- if (!(kt->flags & RELOC_SET))
+ if (!(kt->flags & RELOC_SET)) {
+ if (machine_type("X86_64")) {
+ derive_kaslr_offset(abfd, dynamic, from,
+ fromend, size, store);
+ }
kt->flags |= RELOC_FORCE;
+ }
} else
kt->flags &= ~RELOC_SET;
- first = 0;
- from = (bfd_byte *) minisyms;
- fromend = from + symcount * size;
for (; from < fromend; from += size)
{
if ((sym = bfd_minisymbol_to_symbol(abfd, dynamic, from, store))
--
1.9.0.rc1.175.g0b1dcb5
10 years, 6 months
[PATCH] Automatically detect kernel aslr offset
by Andy Honig
Automatically detext kernel aslr offset
This patch improves support for kernel aslr, to automatically find the
aslr offset based on the location of the _stext symbol in the vmcore
info.
Signed-off-by: Andrew Honig <ahonig(a)google.com>
---
netdump.c | 19 ++++++++-----------
symbols.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++---
2 files changed, 54 insertions(+), 14 deletions(-)
diff --git a/netdump.c b/netdump.c
index 8e7ec15..b327649 100644
--- a/netdump.c
+++ b/netdump.c
@@ -411,18 +411,15 @@ is_netdump(char *file, ulong source_query)
get_log_from_vmcoreinfo(file, vmcoreinfo_read_string);
}
- // This is the code where I should read the aslr offset.
+ /*
+ *We may need the _stext_SYMBOL from the vmcore_info to adjust for
+ * kaslr and we may not have gotten it elsewhere.
+ */
if (source_query == KDUMP_LOCAL) {
- long aslr_offset = 0;
- char *aslr_string = vmcoreinfo_read_string("KERNELOFFSET");
- if (aslr_string) {
- aslr_offset = strtoul(aslr_string, NULL, 16);
- free (aslr_string);
- }
- if (!(kt->flags & RELOC_SET) && aslr_offset > 0) {
- kt->flags |= RELOC_SET;
- kt->relocate=aslr_offset * -1;
- }
+ char *tmpstring = vmcoreinfo_read_string("SYMBOL(_stext)");
+ kt->vmcoreinfo._stext_SYMBOL =
+ htol(tmpstring, RETURN_ON_ERROR, NULL);
+ free(tmpstring);
}
return nd->header_size;
diff --git a/symbols.c b/symbols.c
index d5f8199..afe5ed0 100755
--- a/symbols.c
+++ b/symbols.c
@@ -553,6 +553,43 @@ strip_symbol_end(const char *name, char *buf)
}
/*
+ * Derives the kernel aslr offset by comparing the _stext symbol from the
+ * the vmcore_info in the dump file to the _stext symbol in the vmlinux file.
+ */
+static void
+derive_kaslr_offset(bfd *abfd, int dynamic, bfd_byte *start, bfd_byte *end,
+ unsigned int size, asymbol *store)
+{
+ symbol_info syminfo;
+ asymbol *sym;
+ char *name;
+ unsigned long relocate;
+ char buf[BUFSIZE];
+
+ for (; start < end; start += size) {
+ sym = bfd_minisymbol_to_symbol(abfd, dynamic, start, store);
+ if (sym == NULL)
+ error(FATAL, "bfd_minisymbol_to_symbol() failed\n");
+
+ bfd_get_symbol_info(abfd, sym, &syminfo);
+ name = strip_symbol_end(syminfo.name, buf);
+ if (strcmp("_stext", name) == 0) {
+ relocate = syminfo.value - kt->vmcoreinfo._stext_SYMBOL;
+ /*
+ *To avoid mistaking an mismatched kernel version with
+ * a kaslr offset, we make sure that the offset is
+ * aligned by 0x1000, as it always will be for
+ * kaslr.
+ */
+ if ((relocate & 0x1000) == 0) {
+ kt->relocate = relocate;
+ kt->flags |= RELOC_SET;
+ }
+ }
+ }
+}
+
+/*
* Store the symbols gathered by symtab_init(). The symbols are stored
* in increasing numerical order.
*/
@@ -588,15 +625,21 @@ store_symbols(bfd *abfd, int dynamic, void *minisyms, long symcount,
st->symcnt = 0;
sp = st->symtable;
+ first = 0;
+ from = (bfd_byte *) minisyms;
+ fromend = from + symcount * size;
+
if (machine_type("X86") || machine_type("X86_64")) {
+ /* If kernel aslr offset has not been set, try to guess it. */
+ if (kt->relocate == 0)
+ derive_kaslr_offset(abfd, dynamic, from,
+ fromend, size, store);
+
if (!(kt->flags & RELOC_SET))
kt->flags |= RELOC_FORCE;
} else
kt->flags &= ~RELOC_SET;
- first = 0;
- from = (bfd_byte *) minisyms;
- fromend = from + symcount * size;
for (; from < fromend; from += size)
{
if ((sym = bfd_minisymbol_to_symbol(abfd, dynamic, from, store))
--
1.9.0.rc1.175.g0b1dcb5
10 years, 6 months
earlier patch introducing the --kaslr option
by Dave Anderson
Hi Andy,
I've got a ELF kdump vmcore that was created in-house from a kernel configured
with CONFIG_RANDOMIZE_BASE. I thought I might be able to analyze it by applying
your earlier patch that introduced the --kaslr option. The kernel does not
have the offset registered in the vmcoreinfo, and so I'm trying to determine
the offset, but with no luck.
Earlier, Kees had mentioned this:
>> FWIW, the offset reported during a panic to dmesg is:
>> (unsigned long)&_text - __START_KERNEL
Where does it get reported during a panic exactly? Here's the oops trace, gotten
by running "strings" on the vmcore:
$ strings vmcore
... [ cut ] ...
SysRq : Trigger a crash
"BUG: unable to handle kernel NULL pointer dereference at (null)
"IP: [<ffffffff992bf6cf>] sysrq_handle_crash+0x11/0x1b
PGD 3a067 PUD 2e067 PMD 0
Oops: 0002 [#1] PREEMPT SMP 0"
Modules linked in:
CPU: 0 PID: 1720 Comm: bash Not tainted 3.14.0-rc1+ #1130"
task: ffff88001d028000 ti: ffff88001c986000 task.ti: ffff88001c986000
RIP: 0010:[<ffffffff992bf6cf>] [<ffffffff992bf6cf>] sysrq_handle_crash+0x11/0x1b
RSP: 0018:ffff88001c987e90 EFLAGS: 000100920"
RAX: 000000000000000f RBX: ffffffff9975ed50 RCX: 0000000000000000
RDX: ffff88001d028000 RSI: ffff88001cc0e338 RDI: 0000000000000063
RBP: ffff88001c987e90 R08: 0000000000000002 R09: 0000000000000000
R10: ffffffff994e9630 R11: 0000000000000000 R12: 0000000000000007
R13: 0000000000000246 R14: 0000000000000063 R15: 0000000000000000
FS: 00007f0ec2181740(0000) GS:ffff88001cc00000(0000) knlGS:00000000000000000"
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000001d36f000 CR4: 00000000000006f0
Stack:
ffff88001c987ec8 ffffffff992bfc88 0000000000000002 00007f0ec21870000"
0000000000000002 ffff88001c987f58 0000000000000000 ffff88001c987ee80"
ffffffff992c000c ffff88001c92acc0 00007f0ec2187000 ffff88001c987f080"
Call Trace:
[<ffffffff992bfc88>] __handle_sysrq+0x9b/0x133
[<ffffffff992c000c>] write_sysrq_trigger+0x2d/0x3e
[<ffffffff991681cb>] proc_reg_write+0x45/0x65
[<ffffffff9911897c>] vfs_write+0xbf/0x17c
[<ffffffff9911918f>] SyS_write+0x44/0x7a
[<ffffffff9949ad7d>] system_call_fastpath+0x1a/0x1f0"
Code: 4f 00 00 55 b8 01 00 00 00 48 89 e5 75 07 0f b6 05 b3 20 4f 00 83 e0 01 5d c3 55 c7 05 03 18 61 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 31 c0 c7 05 ba dc 46 00 07 00 0"
"RIP [<ffffffff992bf6cf>] sysrq_handle_crash+0x11/0x1b
RSP <ffff88001c987e90>
CR2: 0000000000000000
ttySffffffff99000000 T _text
UUUU
UUUU
VMCOREINFO
OSRELEASE=3.14.0-rc1+
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffffffff99713250
SYMBOL(node_online_map)=ffffffff997b0c68
SYMBOL(swapper_pg_dir)=ffffffff9970e000
SYMBOL(_stext)=ffffffff990001c8
SYMBOL(vmap_area_list)=ffffffff99745c20
SYMBOL(mem_map)=ffffffff9a1253a8
SYMBOL(contig_page_data)=ffffffff99790000
SYMBOL(mem_section)=ffffffff9a126000
LENGTH(mem_section)=2048
SIZE(mem_section)=16
OFFSET(mem_section.section_mem_map)=0
SIZE(page)=64
SIZE(pglist_data)=53248
SIZE(zone)=12288
SIZE(free_area)=88
SIZE(list_head)=16
SIZE(nodemask_t)=8
OFFSET(page.flags)=0
OFFSET(page._count)=28
OFFSET(page.mapping)=8
OFFSET(page.lru)=32
OFFSET(page._mapcount)=24
OFFSET(page.private)=48
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=49240
OFFSET(pglist_data.node_start_pfn)=49304
OFFSET(pglist_data.node_spanned_pages)=49320
OFFSET(pglist_data.node_id)=49328
OFFSET(zone.free_area)=256
OFFSET(zone.vm_stat)=4280
OFFSET(zone.spanned_pages)=8232
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=8
OFFSET(vmap_area.va_start)=0
OFFSET(vmap_area.list)=48
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=ffffffff9972d290
SYMBOL(log_buf_len)=ffffffff9972d288
SYMBOL(log_first_idx)=ffffffff9a11eb48
SYMBOL(log_next_idx)=ffffffff9a11eb38
SIZE(printk_log)=16
OFFSET(printk_log.ts_nsec)=0
OFFSET(printk_log.len)=8
OFFSET(printk_log.text_len)=10
OFFSET(printk_log.dict_len)=12
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=11
NUMBER(PG_swapcache)=16
NUMBER(PG_slab)=7
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
SYMBOL(phys_base)=ffffffff99713010
SYMBOL(init_level4_pgt)=ffffffff9970e000
CRASHTIME=1391826079
OSRELEASE=3.14.0-rc1+
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffffffff99713250
SYMBOL(node_online_map)=ffffffff997b0c68
SYMBOL(swapper_pg_dir)=ffffffff9970e000
SYMBOL(_stext)=ffffffff990001c8
SYMBOL(vmap_area_list)=ffffffff99745c20
SYMBOL(mem_map)=ffffffff9a1253a8
SYMBOL(contig_page_data)=ffffffff99790000
SYMBOL(mem_section)=ffffffff9a126000
LENGTH(mem_section)=2048
SIZE(mem_section)=16
OFFSET(mem_section.section_mem_map)=0
SIZE(page)=64
SIZE(pglist_data)=53248
SIZE(zone)=12288
SIZE(free_area)=88
SIZE(list_head)=16
SIZE(nodemask_t)=8
OFFSET(page.flags)=0
OFFSET(page._count)=28
OFFSET(page.mapping)=8
OFFSET(page.lru)=32
OFFSET(page._mapcount)=24
OFFSET(page.private)=48
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=49240
OFFSET(pglist_data.node_start_pfn)=49304
OFFSET(pglist_data.node_spanned_pages)=49320
OFFSET(pglist_data.node_id)=49328
OFFSET(zone.free_area)=256
OFFSET(zone.vm_stat)=4280
OFFSET(zone.spanned_pages)=8232
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=8
OFFSET(vmap_area.va_start)=0
OFFSET(vmap_area.list)=48
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=ffffffff9972d290
SYMBOL(log_buf_len)=ffffffff9972d288
SYMBOL(log_first_idx)=ffffffff9a11eb48
SYMBOL(log_next_idx)=ffffffff9a11eb38
SIZE(printk_log)=16
OFFSET(printk_log.ts_nsec)=0
OFFSET(printk_log.len)=8
OFFSET(printk_log.text_len)=10
OFFSET(printk_log.dict_len)=12
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=11
NUMBER(PG_swapcache)=16
NUMBER(PG_slab)=7
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
SYMBOL(phys_base)=ffffffff99713010
SYMBOL(init_level4_pgt)=ffffffff9970e000
CRASHTIME=1391826079
...
Anyway, the /proc/kallsyms file of the crashing system was saved,
and it shows this:
ffffffff99000000 T _text
and if I subtract __START_KERNEL (ffffffff80000000) from that, I get
what I presume is the kaslr offset of 0x19000000. The vmcore core
header would seeminlgy confirm that:
$ readelf -a vmcore
... [ cut ] ...
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
NOTE 0x0000000000001000 0x0000000000000000 0x0000000000000000
0x00000000000007f8 0x00000000000007f8 0
LOAD 0x0000000000002000 0xffffffff99000000 0x0000000019000000 <===
0x0000000001183000 0x0000000001183000 RWE 0
LOAD 0x0000000001185000 0xffff880000001000 0x0000000000001000
0x000000000009f000 0x000000000009f000 RWE 0
LOAD 0x0000000001224000 0xffff880000100000 0x0000000000100000
0x0000000010f00000 0x0000000010f00000 RWE 0
LOAD 0x0000000012124000 0xffff880019000000 0x0000000019000000
0x0000000005194000 0x0000000005194000 RWE 0
LOAD 0x00000000172b8000 0xffff88001e1c1000 0x000000001e1c1000
0x00000000017c0000 0x00000000017c0000 RWE 0
LOAD 0x0000000018a78000 0xffff88001f9e5000 0x000000001f9e5000
0x00000000005fb000 0x00000000005fb000 RWE 0
But if I try that value with your patch applied, it fails in the same manner
as if I don't use the --kaslr option at all:
$ crash --kaslr 0x19000000 vmlinux vmcore
crash 7.0.5rc12
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: could not find MAGIC_START!
WARNING: cannot read linux_banner string
crash: vmlinux and vmcore do not match!
Usage:
crash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
Enter "crash -h" for details.
$
Any ideas? I can give you the vmlinux/vmcore/kallsyms triplet if you'd like.
Thanks,
Dave
10 years, 6 months
Question about: crash: seek error: kernel virtual address: c1625ccc type: "cpu_possible_mask"
by 曾兴胜
Hi,
I am using KDUMP tools(version: 2.0.4) to capture kernel panic information. I configured the kernel compile para according to the kdump-tutorial, and get dump file in /var/crash/ through echo c > /proc/sysrq-trigger. but when I analyze the dump file using command: crash -d8 vmlinux dumpfile
error happens as following:
crash: seek error: kernel virtual address: c1625ccc type: "cpu_possible_mask"
The more log can be seen in the attachment: crash.log
Otherwise, I found the size of dump file(dump-incomplete) is about 25M, but my memory is 4G. why the dump file is so small?
My linux version is linux Mint15, basing on Ubuntu, the kernel version is 3.8.13.
v10@COS-Desktop ~ $
v10@COS-Desktop ~ $ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.8.13.13-cos-i686 root=UUID=c0bd0f62-0dfa-4425-8afa-b5ff8a00df5c ro crashkernel=384M-2G:64M,2G-:128M quiet splash
v10@COS-Desktop ~ $
v10@COS-Desktop ~ $ cat /var/crash/kexec_cmd
/sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-3.8.13.13-cos-i686 root=UUID=c0bd0f62-0dfa-4425-8afa-b5ff8a00df5c ro quiet splash irqpoll maxcpus=1 nousb " --initrd=/boot/initrd.img-3.8.13.13-cos-i686 /boot/vmlinuz-3.8.13.13-cos-i686
v10@COS-Desktop ~ $
v10@COS-Desktop ~ $
other log:
v10@COS-Desktop ~ $ sudo crash --minimal kdump/0211/vmlinux /var/crash/201402120848/dump-incomplete
[sudo] password for v10:
crash 6.1.6
Copyright (C) 2002-2013 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
NOTE: minimal mode commands: log, dis, rd, sym, eval, set, extend and exit
crash>
crash>
crash>rd linux_banner 30
rd: seek error: kernel virtual address: c1622080 type: "32-bit KVADDR"
crash>
crash>
crash>
I appreciate your help. Thanks!
10 years, 6 months
Re: [Crash-utility] Crash-utility Digest, Vol 101, Issue 7
by Dave Anderson
I didn't do it by intention. I did name the release "crash-7.0.5" without realizing it would concatenate the repo name with the release name.
Dave
Sent from my Verizon Wireless 4G LTE smartphone
-------- Original message --------
From: crash-utility-request(a)redhat.com
Date:02/17/2014 12:00 PM (GMT-05:00)
To: crash-utility(a)redhat.com
Subject: Crash-utility Digest, Vol 101, Issue 7
Send Crash-utility mailing list submissions to
crash-utility(a)redhat.com
To subscribe or unsubscribe via the World Wide Web, visit
https://www.redhat.com/mailman/listinfo/crash-utility
or, via email, send a message with subject or body 'help' to
crash-utility-request(a)redhat.com
You can reach the person managing the list at
crash-utility-owner(a)redhat.com
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Crash-utility digest..."
Today's Topics:
1. Re: [ANNOUNCE] crash 7.0.5 is available (Michael Holzheu)
----------------------------------------------------------------------
Message: 1
Date: Mon, 17 Feb 2014 17:44:09 +0100
From: Michael Holzheu <holzheu(a)linux.vnet.ibm.com>
To: "Discussion list for crash utility usage, maintenance and
development" <crash-utility(a)redhat.com>
Subject: Re: [Crash-utility] [ANNOUNCE] crash 7.0.5 is available
Message-ID: <20140217174409.40afcdd4@holzheu>
Content-Type: text/plain; charset=US-ASCII
On Fri, 14 Feb 2014 12:12:46 -0500 (EST)
Dave Anderson <anderson(a)redhat.com> wrote:
>
> Download from: http://people.redhat.com/anderson
> or
> https://github.com/crash-utility/crash/releases
Hello Dave,
Thanks for the repository!
Is it by intention that the tarball from
https://github.com/crash-utility/crash/releases
is "crash-crash-..."?
Best Regards,
Michael
------------------------------
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
End of Crash-utility Digest, Vol 101, Issue 7
*********************************************
10 years, 6 months
[ANNOUNCE] crash 7.0.5 is available
by Dave Anderson
Download from: http://people.redhat.com/anderson
or
https://github.com/crash-utility/crash/releases
Note that the new GitHub repository was initiated with crash-7.0.4.
The master branch will contain all patches that are queued for the
next release:
$ git clone git://github.com/crash-utility/crash.git
Changelog:
- Fix for the "runq -g" option for kernels that are configured with
CONFIG_FAIR_GROUP_SCHED, but not CONFIG_CFS_BANDWIDTH. Without the
patch, the command fails with the message "runq: invalid structure
member offset: cfs_rq_throttled".
(vinayakm.list(a)gmail.com)
- Add support for Xen PVH guest types introduced in Xen 4.4. Without
the patch, running against a Xen 4.4 hypervisor binary would fail
during session initialization with the error message "crash: invalid
structure member offset: domain_is_hvm". In addition, the PVH guest
type is being registered internally as an HVM guest type, the debug
"help -X ofs" command's display of the domain_domain_flags offset
has been fixed to show it in decimal, and the setting of the internal
dc->domain_flags has been fixed to contain all flags set, not just
the first one found.
(dslutz(a)verizon.com)
- Fix for the "kmem -S" command on Linux 3.1 and later kernels that are
configured with CONFIG_SLUB. Because the the page structure's inuse
and objects fields used by SLUB were changed from discrete u16 types
to bit-fields within an unsigned int, the display of per-node partial
slab statistics are incorrect. Without the patch, the TOTAL and
ALLOCATED values are incorrectly shown as equal values, and therefore
the FREE value is always zero.
(anderson(a)redhat.com)
- Fix for the "kmem -S" command for kernels that are configured with
CONFIG_SLUB. Eash per-cpu slab object dump may show incorrect
ALLOCATED and FREE values; and as seen on Linux 3.5 and later
kernels, the TOTAL value and the number of individual objects dumped
may also be incorrect (too small).
(anderson(a)redhat.com)
- When executing the commands from an input file specified by the
"-i <file>" command line option, or when accepting input from a
file as a set of commands or as a set of command arguments using the
"<" redirection character, unconditionally cease the operation if
CTRL-c is entered. Without the patch, depending upon the command
that was running when the SIGINT was received, the operation may
continue uninterruptibly until the file contents are consumed.
(anderson(a)redhat.com)
- Enhanced the "bt -F" option such that if "-F" is entered twice,
and if the stack frame contents reference a slab cache object, both
the slab cache name and the stack contents will be displayed within
brackets.
(anderson(a)redhat.com)
- Enhanced the "rd -S" option such that if "-S" is entered twice,
and if the memory contents reference a slab cache object, both the
slab cache name and the memory contents will be displayed within
brackets.
(anderson(a)redhat.com)
- Fix for the X86_64 "bt" command to prevent an unwarranted message
indicating "WARNING: possibly bogus exception frame" generated
from a blocked kernel thread that was in the process of exec'ing
a user process via the call_usermodehelper() facility.
(anderson(a)redhat.com)
- Fix for the X86_64 "bt" command to more correctly determine the
function frame that called into an interrupted function. Without
the patch, the first frame just above an IRQ exception frame
register dump may show an invalid/stale function.
(anderson(a)redhat.com)
- Fix for the X86_64 "bt" command if a page fault exception was
generated by the invalid contents of the RIP register. Without
the patch, the exception frame register dump is not displayed
above the "page_fault" stack frame; and in a related issue, the
"bt -e" option will not find and display the exception frame.
(anderson(a)redhat.com)
- When invoking a crash session with a compressed vmlinux file,
make the same host-machine/vmlinux endian verification that is
done with uncompressed vmlinx files.
(anderson(a)redhat.com)
- Reduce the number of CTRL-c entries required to unconditionally
terminate any manually-entered command from three to one.
(anderson(a)redhat.com)
- Fix for the X86_64 "bt" command if an async page fault exception
occurred in a KVM guest running a Linux 2.6.38 or later kernel.
Without the patch, the exception frame register dump is not displayed
above the "async_page_fault" stack frame.
(anderson(a)redhat.com)
10 years, 6 months
Re: [Crash-utility] Crash, won't read my vmcore "crash: page excluded: kernel virtual address:"
by Tory M Blue
Dave Anderson reached out and wrote:
----- Original Message -----
> [root kvm7 127.0.0.1-2014-02-07-19:17:09]# crash
/boot/System.map-2.6.32-220.el6.x86_64.debug
/usr/lib/debug/lib/modules/2.6.32-220.el6.x86_64.debug/vmlinux vmcore
>
> crash 5.1.8-1.el6
> Copyright (C) 2002-2011 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
details.
> GNU gdb (GDB) 7.0
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html
> >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> crash: page excluded: kernel virtual address: ffffffff81542000 type:
"cpu_possible_mask"
>
> I can go into minimal,
>
>
> nm -Bn /usr/lib/debug/lib/modules/2.6.32-220.el6.x86_64.debug/vmlinux |
grep _stext
> ffffffff81000198 T _stext
>
> cat /proc/kallsyms | grep _stext
> ffffffff81000198 T _stext
>
> If I use the System Map parm I get this warning
>
> WARNING: kernels compiled by different gcc versions:
> /usr/lib/debug/lib/modules/2.6.32-220.el6.x86_64.debug/vmlinux: 4.4.5
> vmcore kernel: 4.4.6
>
>
> Would really like to understand why this system crashed. I know I'm a bit
> behind on my kernel versions however, but I should be able to look at this
> kernel??
>
> Thanks
> Tory
It looks like the vmcore and vmlinux file don't match, like maybe the
crashing
system was running the standard 2.6.32-220.el6.x86_64 kernel, and you're
trying
to debug it using the 2.6.32-220.el6.x86_64.debug kernel variant?
First thing -- *never* use a System.map file unless for some reason you
don't
have the original kernel's vmlinux available *and* you feel that the vmlinux
file you have is very close to the crashing kernel's vmlinux. Bit with any
RHEL standard (unmodified) vmlinux/vmcore setup, the System.map is
completely
useless.
So the first question is: what kernel generated the vmcore?
Do this:
$ strings vmcore | grep '2.6.32'
Dave
--
Dave you are right, I thought I had to use the devel kernel and in fact my
system is not running that, so it crashed with the standard
2.6.32-220.el6.x86_64 kernel.
[tblue@kvm7 127.0.0.1-2014-02-07-19:17:09]$ sudo strings vmcore | grep
'2.6.32'
2.6.32-220.el6.x86_64
OSRELEASE=2.6.32-220.el6.x86_64
But it won't take my vmlinux from /boot
crash: /boot/vmlinuz-2.6.32-220.el6.x86_64: not a supported file format
Yes sir you were correct, I was using the wrong kernel!
please wait... (determining panic task)
WARNING: multiple active tasks have called die
KERNEL: /usr/lib/debug/lib/modules/2.6.32-220.el6.x86_64/vmlinux
DUMPFILE: /libvirt/crash/127.0.0.1-2014-02-07-19:17:09/vmcore [PARTIAL
DUMP]
CPUS: 32
DATE: Fri Feb 7 18:16:05 2014
UPTIME: 226 days, 21:36:13
LOAD AVERAGE: 2.42, 2.68, 2.69
TASKS: 816
NODENAME: kvm7.domain.com
RELEASE: 2.6.32-220.el6.x86_64
VERSION: #1 SMP Tue Dec 6 19:48:22 GMT 2011
MACHINE: x86_64 (2200 Mhz)
MEMORY: 88 GB
PANIC: ""
PID: 0
COMMAND: "swapper"
TASK: ffff881665514b40 (1 of 32) [THREAD_INFO: ffff880c6124e000]
CPU: 19
STATE: TASK_RUNNING (PANIC)
Nothing stands out as s bug or reason to fail
divide error: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/cpu31/cache/index2/shared_cpu_map
CPU 19
Modules linked in: ext3 jbd ip6table_filter ip6_tables ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables
sunrpc bridge stp llc bonding ipv6 vhost_net macvtap macvlan tun kvm_intel
kvm cdc_ether usbnet mii microcode i2c_i801 i2c_core iTCO_wdt
iTCO_vendor_support shpchp igb ioatdma dca ses enclosure sg ext4 mbcache
jbd2 sr_mod cdrom sd_mod crc_t10dif ahci megaraid_sas dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 0, comm: swapper Not tainted 2.6.32-220.el6.x86_64 #1 IBM System x3650
M4 -[7915AC1]-/00J6528
RIP: 0010:[<ffffffff81054ad5>] [<ffffffff81054ad5>]
find_busiest_group+0x5c5/0xb20
RSP: 0018:ffff880028363c40 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff880028363e64 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8800282cf540 RDI: ffff8800282d5fc0
RBP: ffff880028363dd0 R08: ffff8800282cf860 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000001 R12: 00000000ffffff01
R13: 0000000000015fc0 R14: ffffffffffffffff R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff880028360000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007f4e5215c000 CR3: 00000011bea54000 CR4: 00000000000426e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff880c6124e000, task ffff881665514b40)
Stack:
ffff880028363d70 ffff880028363ce0 ffff880028363ca0 000000000000024d
<0> ffff8800282cf860 ffff880028363e58 0101881664b121a8 0000000600000000
<0> 0000000600000000 ffff8800282cf540 0000000123386cc0 0000000000000008
Call Trace:
<IRQ>
[<ffffffffa02e4669>] ? br_handle_frame_finish+0x179/0x2a0 [bridge]
[<ffffffff8105fc52>] rebalance_domains+0x1a2/0x5b0
[<ffffffff81060153>] run_rebalance_domains+0xf3/0x160
[<ffffffff8107c4f0>] ? get_next_timer_interrupt+0x1b0/0x250
[<ffffffff81072161>] __do_softirq+0xc1/0x1d0
[<ffffffff81097e0a>] ? sched_clock_idle_wakeup_event+0x1a/0x20
[<ffffffff8100c24c>] call_softirq+0x1c/0x30
[<ffffffff8100de85>] do_softirq+0x65/0xa0
[<ffffffff81071f45>] irq_exit+0x85/0x90
[<ffffffff8102a255>] smp_call_function_single_interrupt+0x35/0x40
[<ffffffff8100bdb3>] call_function_single_interrupt+0x13/0x20
<EOI>
[<ffffffff812c4a5e>] ? intel_idle+0xde/0x170
[<ffffffff812c4a41>] ? intel_idle+0xc1/0x170
[<ffffffff813f9f47>] cpuidle_idle_call+0xa7/0x140
[<ffffffff81009e06>] cpu_idle+0xb6/0x110
[<ffffffff814e5f23>] start_secondary+0x202/0x245
Code: d0 b8 01 00 00 00 48 c1 ea 0a 48 85 d2 0f 45 c2 41 89 40 08 66 90 4c
8b 85 e0 fe ff ff 48 8b 45 a8 31 d2 41 8b 48 08 48 c1 e0 0a <48> f7 f1 48
8b 4d b0 48 89 45 a0 31 c0 48 85 c9 74 0c 48 8b 45
RIP [<ffffffff81054ad5>] find_busiest_group+0x5c5/0xb20
RSP <ffff880028363c40>
Is there a forum that would help me figure out what exactly cause this
crash as it's not the first time, across this series of servers running KVM
Thank you sir,
Tory
10 years, 7 months