crash version 4.0-8.10 is available
by Dave Anderson
- Enhancement for currently-existing "mod -S <directory>" option to
make it search for the module debuginfo tree in the same specified,
non-standard, directory tree. When "mod -S" is used without a
specified directory argument, the "<module>.ko" object files are
searched for in the standard "/lib/modules/<release>" directory
tree, and their associated "<module>.ko.debug" are searched for
in the standard "/usr/lib/debug/lib/modules/<release>" directory
tree. Without this patch, "mod -S <directory>" would search the
specified non-standard directory tree for the kernel's "<module>.ko"
files, but the associated "<module>.ko.debug" would not be found.
With the patch, the search for the associated "<module>.ko.debug"
files will be made in the following order and manner:
1. in the same directory containing the "<module>.ko" file.
2. in the ".debug" subdirectory of the directory containing the
"<module>.ko" file.
3. if the "<module>.ko" file was found in a directory pathname
containing the "/lib/modules" component, then the search will be
made in the assocated "/usr/lib/debug/lib/modules" location.
This enhancement will allow an alternate module/module-debuginfo
directory tree to be set up like so:
# cd <directory>
# rpm2cpio kernel-<release>.rpm | cpio -idv
# rpm2cpio kernel-debuginfo-<release>.rpm | cpio -idv
Having done that, the currently-existing "mod -S <directory>"
option will find both the "<module>.ko" and "<module>.ko.debug"
files. In addition, a new "--mod" command-line option may be used
to specify the directory tree:
# crash vmlinux [vmcore] --mod <directory>
When the "--mod <directory>" command line option is used, then
"mod -S" (without a directory argument) will search that directory
tree by default instead of using the standard location.
(anderson(a)redhat.com)
- Fix to handle the 2.6.29 replacement of the symbols "cpu_online_map",
"cpu_present_map" and "cpu_possible_map" with analogous symbols
"cpu_online_mask", "cpu_present_mask" and "cpu_possible_mask".
Without this patch, crash would fail during initialization on s390
and s390x systems with the error message "crash: cannot resolve
cpu_online_map", or with the error message "crash: PPC64: cannot find
cpu_present_map or cpu_online_map symbols" on ppc64 systems.
(holzheu(a)linux.vnet.ibm.com, anderson(a)redhat.com)
- Added several function prototypes for the SIAL extension module file
sial.c because of its inability to #include "defs.h". Without
the patch, the compiler would presume that several un-prototyped
functions would have a return value of int, and therefore would
truncate 64-bit return values into 32-bits.
(holzheu(a)linux.vnet.ibm.com)
- If by remote chance the panic task cannot be determined from a ppc64
kdump vmcore, a segmentation violation would occur during crash
session initialization.
(anderson(a)redhat.com)
- An additional directory has been added to the currently-existing
list of directories that the "extend" command searches when the
extension module file is not expressed with a fully-qualified
pathname. The following directories will be searched in the order
shown, and the first instance of the file that is found will be
selected:
1. the current working directory
2. the directory specified in the CRASH_EXTENSIONS shell
environment variable
3. /usr/lib64/crash/extensions (64-bit architectures)
4. /usr/lib/crash/extensions
5. ./extensions
(anderson(a)redhat.com)
- Fix the "extensions/Makefile" to force a rebuild of extension modules
when the "defs.h" file is newer than the module source.
(anderson(a)redhat.com)
- Added "snap.c" and "snap.mk" files to the extensions directory. The
new module contains a "snap" command that creates a kdump or netdump
dumpfile from a live system. Currently the x86, x86_64, ppc64 and
ia64 architectures are supported. The snap.so extension module has
been added to the crash-extensions-<release>.rpm, which is created
by the crash-<release>.src.rpm, which installs extension modules
in the /usr/lib[64]/crash/extensions directory.
(anderson(a)redhat.com)
- Added a set of functions that, for an active task, return a pointer
to the associated register set found in an NT_PRSTATUS note in
netdump and kdump ELF dumpfiles if one exists. They are not used by
the crash source code, but are available to extension modules.
(sharyath(a)in.ibm.com)
- Use the "crashing_cpu" kernel symbol as a more efficient manner of
determining a kdump x86_64 panic task.
(anderson(a)redhat.com)
- Fix to handle the replacement of the per-cpu array of x8664_pda data
structures with per-cpu variables in 2.6.30. Without the patch, an
x86_64 crash session would die during initialization with the error
message: "crash: invalid structure size: x8664_pda".
(anderson(a)redhat.com, nishimura(a)mxp.nes.nec.co.jp)
Download from: http://people.redhat.com/anderson
15 years, 1 month
Re: crash 4.0-8.9 w/ 2.6.30-rc6
by Dave Anderson
Hello Daisuke,
I merged your patch with mine, and have things working relatively well.
One other issue comes up -- when the percpu data areas are set up, they
are done with a "for_each_possible_cpu(cpu)", so your new x86_64_per_cpu_init()
function (may) return more cpus than are actually online. At least that's
the case with Mike's vmcore -- it's running with 2 cpus online, but the
percpu data area was set up for 4. Prior to 2.6.30, it appears that only online
cpus initialized their x8664_pda structures. Anyway, I guess we'll just
have to scale back the cpu count based upon the contents of the cpu_online_map.
I changed the x86_64_verify_symbol() function to look like this so as to only
collect the __per_cpu_start/end and per_cpu__xxx symbols:
/*
* Accept or reject a symbol from the kernel namelist.
*/
static int
x86_64_verify_symbol(const char *name, ulong value, char type)
{
if (!name || !strlen(name))
return FALSE;
if (!(machdep->flags & KSYMS_START)) {
if (STREQ(name, "_text") || STREQ(name, "_stext")) {
machdep->flags |= KSYMS_START;
if (!st->first_ksymbol)
st->first_ksymbol = value;
return TRUE;
} else if (STREQ(name, "__per_cpu_start")) {
st->flags |= PERCPU_SYMS;
return TRUE;
} else if ((st->flags & PERCPU_SYMS) &&
(STRNEQ(name, "per_cpu") || STRNEQ(name, "__per_cpu")))
return TRUE;
else
return FALSE;
}
return TRUE;
}
where the PERCPU_SYMS flag is held in the global symbol_table_data flags
field, and where it gets used like this in symbols.c:in_ksymbol_range():
/*
* Determine whether an address falls within the kernel's, or any module's,
* address space.
*/
int
in_ksymbol_range(ulong value)
{
if ((value >= st->symtable[0].value) &&
(value <= st->symtable[st->symcnt-1].value)) {
if ((st->flags & PERCPU_SYMS) && (value < st->first_ksymbol))
return FALSE;
else
return TRUE;
}
if (module_symbol(value, NULL, NULL, NULL, output_radix))
return TRUE;
if (machdep->value_to_symbol(value, NULL))
return TRUE;
return FALSE;
}
This seems to (so far) clear up the inadvertent dumping of "false" value-to-symbol
translations in situations like "rd -s", but allows that usage of the per_cpu_xxx
symbols are arguments to symbol_value(), symbol_exists(), symbol_search(), etc.
Other than that, so far I've left the rest of your patch intact, and I wanted
to thank you again for the contribution.
Dave
15 years, 1 month
Re: crash 4.0-8.9 w/ 2.6.30-rc6
by Dave Anderson
----- "Daisuke Nishimura" <nishimura(a)mxp.nes.nec.co.jp> wrote:
> Hi, Dave.
>
> It's a very and very dirty hack, but some of my colleagues say that
> a dump file of 2.6.30-rcX can be analyzed by this patch.
>
> I hope it would be some help for you.
>
> Thanks,
> Daisuke Nishimura.
Actually it's not so dirty at all -- it's pretty well done!
I was also tinkering with a few new schemes yesterday, and did
make it to the "crash> " prompt as well. But I was allowing the
kt->__per_cpu_offset[] array to be initialized in kernel_init()
(line 195) as is done for x86, althougyh I think now that I prefer
the way that you guys have done it.
I also was trying to come up with a clean way to deal with
the x86_64 "per_cpu__xxx" symbols being contained in the
in the crash kernel symbol list. With my test crash -- and
I presume the same with your patch -- the symbol list
look like this:
crash> sym -l
0 (D) __per_cpu_start
0 (D) per_cpu__irq_stack_union
4000 (D) per_cpu__gdt_page
5000 (d) per_cpu__exception_stacks
b000 (D) per_cpu__current_task
b008 (d) per_cpu__is_idle
b010 (D) per_cpu__old_rsp
b018 (D) per_cpu__irq_regs
b020 (D) per_cpu__vector_irq
b420 (d) per_cpu__cpu_devices
b478 (D) per_cpu__cyc2ns
b480 (d) per_cpu__cpuid4_info
b488 (d) per_cpu__cache_kobject
b490 (d) per_cpu__index_kobject
b4a0 (D) per_cpu__irq_stack_ptr
b4a8 (D) per_cpu__kernel_stack
b4b0 (D) per_cpu__irq_count
b4c0 (D) per_cpu__orig_ist
... [ snip ] ...
10540 (D) per_cpu__irq_stat
10580 (D) per_cpu__cpu_info
10680 (D) per_cpu__cpu_tlbstate
106c0 (d) per_cpu__runqueues
10fc0 (d) per_cpu__sched_clock_data
10fe0 (D) __per_cpu_end
ffffffff80200000 (T) _text
ffffffff80200000 (T) startup_64
ffffffff802000b7 (t) ident_complete
...
The problem with having symbols below "_text" in this case
is that commands like "rd -s" or other memory dumps that
do symbolic translations of numeric values run into the
per_cpu symbols. So for example, a page of zeroes would
look like this:
crash> rd empty_zero_page 10
ffffffff80833000: 0000000000000000 0000000000000000 ................
ffffffff80833010: 0000000000000000 0000000000000000 ................
ffffffff80833020: 0000000000000000 0000000000000000 ................
ffffffff80833030: 0000000000000000 0000000000000000 ................
ffffffff80833040: 0000000000000000 0000000000000000 ................
crash> rd -s empty_zero_page 10
ffffffff80833000: per_cpu__irq_stack_union per_cpu__irq_stack_union
ffffffff80833010: per_cpu__irq_stack_union per_cpu__irq_stack_union
ffffffff80833020: per_cpu__irq_stack_union per_cpu__irq_stack_union
ffffffff80833030: per_cpu__irq_stack_union per_cpu__irq_stack_union
ffffffff80833040: per_cpu__irq_stack_union per_cpu__irq_stack_union
crash>
In fact, *any* value would translate to a symbol:
crash> rd ffff88007e4ee820 10
ffff88007e4ee820: 0000000000400000 0000000000409000 ..@.......@.....
ffff88007e4ee830: ffff88007e4eead8 0000000000000025 ..N~....%.......
ffff88007e4ee840: 0000000008001875 ffff88007e4eeb09 u.........N~....
ffff88007e4ee850: 0000000000000000 0000000000000000 ................
ffff88007e4ee860: ffff88007e4ee860 ffff88007e4ee860 `.N~....`.N~....
crash> rd -s ffff88007e4ee820 10
ffff88007e4ee820: __per_cpu_end+4124704 __per_cpu_end+4161568
ffff88007e4ee830: __per_cpu_end+-131939276301576 per_cpu__irq_stack_union+37
ffff88007e4ee840: __per_cpu_end+134154389 __per_cpu_end+-131939276301527
ffff88007e4ee850: per_cpu__irq_stack_union per_cpu__irq_stack_union
ffff88007e4ee860: __per_cpu_end+-131939276302208 __per_cpu_end+-131939276302208
crash>
So clearly there will have to be special handling for those
symbols, because sometimes a caller will *want* to receive
information re: those symbols, and other times will not.
(i.e. in symbol_exists(), value_search(), etc.) It was
never a problem in the past because the per_cpu__xxx
symbols were not "real" symbol values, but they were
very large numberic values in the kernel symbol list.
So in any case, we're on the same page, and I do like
what you've done. Thank you very much for having already
looked into this!
Dave
>
> Signed-off-by: Daisuke Nishimura <nishimura(a)mxp.nes.nec.co.jp>
> ---
> defs.h | 1 +
> x86_64.c | 78
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----
> 2 files changed, 74 insertions(+), 5 deletions(-)
>
> diff --git a/defs.h b/defs.h
> index e5e3538..32b2e1e 100755
> --- a/defs.h
> +++ b/defs.h
> @@ -3851,6 +3851,7 @@ struct machine_specific {
> #define PHYS_BASE (0x80)
> #define VM_XEN_RHEL4 (0x100)
> #define VMEMMAP (0x200)
> +#define KSYMS_PERCPU (0x400)
>
> #define VM_FLAGS (VM_ORIG|VM_2_6_11|VM_XEN|VM_XEN_RHEL4)
>
> diff --git a/x86_64.c b/x86_64.c
> index 993827e..4813f96 100755
> --- a/x86_64.c
> +++ b/x86_64.c
> @@ -71,6 +71,7 @@ static int x86_64_is_uvaddr(ulong, struct
> task_context *);
> void x86_64_compiler_warning_stub(void);
> static void x86_64_init_kernel_pgd(void);
> static void x86_64_cpu_pda_init(void);
> +static void x86_64_per_cpu_init(void);
> static void x86_64_ist_init(void);
> static void x86_64_post_init(void);
> static void parse_cmdline_arg(void);
> @@ -290,7 +291,10 @@ x86_64_init(int when)
> MEMBER_OFFSET_INIT(user_regs_struct_ss,
> "user_regs_struct", "ss");
> STRUCT_SIZE_INIT(user_regs_struct, "user_regs_struct");
> - x86_64_cpu_pda_init();
> + if (STRUCT_EXISTS("x8664_pda"))
> + x86_64_cpu_pda_init();
> + else
> + x86_64_per_cpu_init();
> x86_64_ist_init();
> if ((machdep->machspec->irqstack = (char *)
> malloc(machdep->machspec->stkinfo.isize)) == NULL)
> @@ -677,6 +681,51 @@ x86_64_cpu_pda_init(void)
> FREEBUF(cpu_pda_buf);
> }
>
> +static void
> +x86_64_per_cpu_init(void)
> +{
> + int i, cpus, cpunumber;
> + ulong istacksize;
> +
> + if (!(kt->flags & PER_CPU_OFF))
> + return;
> +
> + if (!symbol_exists("per_cpu__cpu_number") || !symbol_exists("per_cpu__irq_stack_ptr"))
> + return;
> +
> + for (i = cpus = 0; i < NR_CPUS; i++) {
> + readmem(symbol_value("per_cpu__cpu_number") + kt->__per_cpu_offset[i],
> + KVADDR, &cpunumber, sizeof(int),
> + "cpu number (per_cpu)", FAULT_ON_ERROR);
> + if (cpunumber != cpus)
> + break;
> + cpus++;
> +
> + readmem(symbol_value("per_cpu__irq_stack_ptr") + kt->__per_cpu_offset[i],
> + KVADDR, &machdep->machspec->stkinfo.ibase[i],
> + sizeof(ulong), "irq stack ptr (per cpu)", FAULT_ON_ERROR);
> + }
> +
> + istacksize = 16384; /* 16K */
> + machdep->machspec->stkinfo.isize = istacksize;
> +
> + /*
> + * Adjust the kernel top-of-stack values down to their base.
> + */
> + for (i = 0; i < NR_CPUS; i++) {
> + if (machdep->machspec->stkinfo.ibase[i])
> + machdep->machspec->stkinfo.ibase[i] -= (istacksize-64);
> + else
> + break;
> + }
> +
> + kt->cpus = cpus;
> + if (kt->cpus > 1)
> + kt->flags |= SMP;
> +
> + verify_spinlock();
> +}
> +
> /*
> * Gather the ist addresses for each CPU.
> */
> @@ -754,7 +803,7 @@ x86_64_ist_init(void)
> */
> sp = value_search(ms->stkinfo.ebase[0][0], &offset);
> if (!sp || offset || !STREQ(sp->name,
> "boot_exception_stacks")) {
> - if (symbol_value("boot_exception_stacks")) {
> + if (symbol_exists("boot_exception_stacks")) {
> error(WARNING,
> "cpu 0 first exception stack: %lx\n
> boot_exception_stacks: %lx\n\n",
> ms->stkinfo.ebase[0][0],
> @@ -1706,8 +1755,12 @@ x86_64_verify_symbol(const char *name, ulong
> value, char type)
> {
> if (STREQ(name, "_text") || STREQ(name, "_stext"))
> machdep->flags |= KSYMS_START;
> + if (STREQ(name, "__per_cpu_start"))
> + machdep->flags |= KSYMS_PERCPU;
> + if (STREQ(name, "__per_cpu_end"))
> + machdep->flags &= ~KSYMS_PERCPU;
>
> - if (!name || !strlen(name) || !(machdep->flags & KSYMS_START))
> + if (!name || !strlen(name) || !(machdep->flags & (KSYMS_START | KSYMS_PERCPU)))
> return FALSE;
> return TRUE;
> }
> @@ -3943,8 +3996,23 @@ x86_64_get_smp_cpus(void)
> char *cpu_pda_buf;
> ulong level4_pgt, cpu_pda_addr;
>
> - if (!VALID_STRUCT(x8664_pda))
> - return 1;
> + if (!VALID_STRUCT(x8664_pda)) {
> + if (!(kt->flags & PER_CPU_OFF))
> + return 1;
> +
> + if (!symbol_exists("per_cpu__cpu_number"))
> + return 1;
> +
> + for (i = cpus = 0; i < NR_CPUS; i++) {
> + readmem(symbol_value("per_cpu__cpu_number") + kt->__per_cpu_offset[i],
> + KVADDR, &cpunumber, sizeof(int),
> + "cpu number (per_cpu)",FAULT_ON_ERROR);
> + if (cpunumber != cpus)
> + break;
> + cpus++;
> + }
> + return cpus;
> + }
>
> cpu_pda_buf = GETBUF(SIZE(x8664_pda));
>
15 years, 1 month
Re: Cannot open dump when using 2G/2G memory split with 2.6.27.21 SMP x86 kernel
by Tero Pirkkanen
Hi
Thanks for your quick answer.
On Wed, May 27, 2009 at 4:49 PM, Dave Anderson <anderson(a)redhat.com> wrote:
> unity-mapped addresses and the data that is being read is obviously
> junk (presuming that you don't have 32 online cpus):
Yes. Junk it is.
> So it looks to be a miscalculation of machdep->kvbase.
I added a line that prints the kvbase:
printf("** kvbase: %x (%x)\n", machdep->kvbase, symbol_value("_stext"));
It produced a value, that to me seems to be correct:
** kvbase: 80000000 (80201000)
My kernel configuration says:
# zcat /proc/config.gz |grep PAGE_OFFSET
CONFIG_PAGE_OFFSET=0x80000000
Also the UP kernel with 2G split (which works) have these same values.
> If you take the same dumpfile above, but use -d8 instead of -d7, you'll
> also see the translated physical address after each "readmem" line
> above.
Couple of first symbols:
<readmem: 8053f700, KVADDR, "kernel_config_data", 32768, (ROE), 97ab7f8>
addr: 8053f700 paddr: 53f700 cnt: 2304
addr: 80540000 paddr: 540000 cnt: 4096
addr: 80541000 paddr: 541000 cnt: 4096
addr: 80542000 paddr: 542000 cnt: 4096
addr: 80543000 paddr: 543000 cnt: 4096
addr: 80544000 paddr: 544000 cnt: 4096
addr: 80545000 paddr: 545000 cnt: 4096
addr: 80546000 paddr: 546000 cnt: 4096
addr: 80547000 paddr: 547000 cnt: 1792
WARNING: could not find MAGIC_START!
GETBUF(128 -> 0)
FREEBUF(0)
GETBUF(4 -> 0)
<readmem: 807348c0, KVADDR, "cpu_possible_map", 4, (ROE), 8430680>
addr: 807348c0 paddr: 7348c0 cnt: 4
...
When I compare the address to addresses from running system, at least
the addresses from symbol table seems to be right. I have compiled the
kallsyms in:
# cat /proc/kallsyms |grep kernel_config_data
8053e700 r kernel_config_data
# cat /proc/kallsyms |grep cpu_possible_map
8065f4a4 r __ksymtab_cpu_possible_map
80666d26 r __kstrtab_cpu_possible_map
807328c0 B cpu_possible_map
...
Also the symbol from the vmlinux image containing debugging symbols is correct:
# objdump -t s3/boot/vmlinux-2.6.27.21+smp-syms |grep kernel_config_data
8053f700 l O .rodata 00002b3a kernel_config_data
Any suggestion how to continue would be appreciated...
Thank you again,
Tero
15 years, 1 month
Re: crash 4.0-8.9 w/ 2.6.30-rc6
by Dave Anderson
----- "Mike Snitzer" <snitzer(a)redhat.com> wrote:
> Hi Dave,
>
> crash is failing with the following when I try to throw a 2.6.30-rc6
> vmcore at it:
>
> crash: invalid structure size: x8664_pda
> FILE: x86_64.c LINE: 584 FUNCTION: x86_64_cpu_pda_init()
>
> [/usr/bin/crash] error trace: 449c7f => 4ce815 => 4d00cf => 50936d
>
> 50936d: SIZE_verify+168
> 4d00cf: (undetermined)
> 4ce815: x86_64_init+3205
> 449c7f: main_loop+152
>
> I can dig deeper but your help would be very much appreciated.
>
> Mike
The venerable "been-there-since-the-beginning-of-x86_64" x8664_pda
data structure no longer exists. It was a per-cpu array of a fundamental
data structure that things like "current", the per-cpu magic number, the
cpu number, the current kernel stack pointer, the per-cpu IRQ stack pointer,
etc. all came from:
/* Per processor datastructure. %gs points to it while the kernel runs */
struct x8664_pda {
struct task_struct *pcurrent; /* Current process */
unsigned long data_offset; /* Per cpu data offset from linker address */
unsigned long kernelstack; /* top of kernel stack for current */
unsigned long oldrsp; /* user rsp for system call */
#if DEBUG_STKSZ > EXCEPTION_STKSZ
unsigned long debugstack; /* #DB/#BP stack. */
#endif
int irqcount; /* Irq nesting counter. Starts with -1 */
int cpunumber; /* Logical CPU number */
char *irqstackptr; /* top of irqstack */
int nodenumber; /* number of current node */
unsigned int __softirq_pending;
unsigned int __nmi_count; /* number of NMI on this CPUs */
int mmu_state;
struct mm_struct *active_mm;
unsigned apic_timer_irqs;
} ____cacheline_aligned_in_smp;
There have been upstream rumblings about replacing it with a more efficient
per-cpu implementation for some time now, but I haven't studied how the new
scheme works yet. It will be a major re-work for the crash utility, so you're
pretty much out of luck for now. (Try "gdb vmlinux vmcore" for basic info)
In the meantime, can you give me a copy of your vmcore? (offline -- note that
I'm forwarding this to the crash-utility mailing list). And I'll start working
on it.
Thanks,
Dave
15 years, 1 month
Re: [PATCH 0/3] Display local variables & function parameters from stack frames
by Dave Anderson
Sharyathi Nagesh wrote:
> Dave
>
> Excuse me for overlooking this part of the code, I am attaching a fix to this, hope this fixes the issue.
I cleaned up your latest version a bit, and have attached the proposed patch
to defs.h and netdump.c. The only functional differences I made was to ensure
that there are multiple NT_PRSTATUS sections when the requested task is just an
active task (i.e. not the panic task). You had correctly made that check for ppc64,
but not for the other two arches.
Dave
15 years, 1 month
Cannot open dump when using 2G/2G memory split with 2.6.27.21 SMP x86 kernel
by Tero Pirkkanen
Hello
I've been trying to open crashdump produced by 2.6.27.21 SMP kernel
with 2G memory split without luck.
I tried to generate the dump by copying from /proc/vmcore and with
makedumpfile 1.3.3. Both without success.
The crash works fine with uniprocessor kernel as well as SMP kernel
using standard memory split. Architecture is 32bit x86 and the crash
was generated inside vmware server 2.0.
Is this supported configuration? Have anybody tried crash dump with 2G
split in recent 2.6.x kernels?
I'll attach some debug output from both non-working smp dump and
working UP dump.
The kernel with debug symbols uses different flavor in name (smp-syms)
than the image without those (smp), so crash always warns that kernel
and dump do not match.
------- output from non-working SMP dump (dumped directly from /proc/vmcore)
# crash -d7 ....
crash 4.0-8.9
....
vmcore_data:
flags: a0 (KDUMP_LOCAL|KDUMP_ELF32)
ndfd: 3
ofp: f7eba4c0
header_size: 540
num_pt_load_segments: 4
pt_load_segment[0]:
file_offset: 21c
phys_start: 0
phys_end: a0000
zero_fill: 0
pt_load_segment[1]:
file_offset: a021c
phys_start: 100000
phys_end: 1000000
zero_fill: 0
pt_load_segment[2]:
file_offset: fa021c
phys_start: 2800000
phys_end: fef0000
zero_fill: 0
pt_load_segment[3]:
file_offset: e69021c
phys_start: ff00000
phys_end: 10000000
zero_fill: 0
elf_header: 9a266f8
elf32: 9a266f8
notes32: 9a2672c
load32: 9a2674c
elf64: 0
notes64: 0
load64: 0
nt_prstatus: 9a267cc
nt_prpsinfo: 0
nt_taskstruct: 0
task_struct: 0
page_size: 0
switch_stack: 0
xen_kdump_data: (unused)
num_prstatus_notes: 2
vmcoreinfo: 0
size_vmcoreinfo: 0
nt_prstatus_percpu:
09a267cc 09a26870
.... skip to end
<readmem: 8053f700, KVADDR, "kernel_config_data", 32768, (ROE), a5f17c0>
WARNING: could not find MAGIC_START!
<readmem: 807348c0, KVADDR, "cpu_possible_map", 4, (ROE), 8430680>
cpu_possible_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30 31
<readmem: 806c478c, KVADDR, "cpu_present_map", 4, (ROE), 8430680>
cpu_present_map: 3 7 8 9 10 14 18 19 21 22 31
<readmem: 806c42a0, KVADDR, "cpu_online_map", 4, (ROE), 8430680>
cpu_online_map: 8 9 14 18 19 21 22 31
<readmem: 807506e0, KVADDR, "xtime", 8, (FOE), 8408d38>
<readmem: 80679e84, KVADDR, "init_uts_ns", 390, (ROE), 840931c>
crash: cannot determine base kernel version
<readmem: 8053a000, KVADDR, "accessible check", 4, (ROE|Q), ff96df04>
<readmem: 8053a000, KVADDR, "readstring characters", 1499, (ROE|Q), ff96cf00>
linux_banner:
&y �&y �&y �(z �]z �jz ��z �a� �f� ��� � � ��� �9� �� �0� �6� �<� �Њ
�B� �H� �N� �
crash: s3/boot/vmlinux-2.6.27.21+smp-syms and
kerneldump-20090526-140502 do not match
------- output from working UP crash (this was generated with makedumpfile)
compressed kdump: header->utsname.machine:
diskdump_data:
filename: (null)
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
dfd: 3
ofp: 0
machine_type: 3 (EM_386)
header: 94c96f8
signature: "KDUMP "
header_version: 2
utsname:
sysname:
nodename:
release:
version:
machine:
domainname:
timestamp:
tv_sec: 0
tv_usec: 0
status: 0 ()
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 4
max_mapnr: 65536
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 1
tasks[nr_cpus]: 0
sub_header: 0 (n/a)
sub_header_kdump: 94ca700
phys_base: 0
dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
data_offset: 6000
block_size: 4096
block_shift: 12
bitmap: 94cb708
bitmap_len: 16384
dumpable_bitmap: 94cf710
byte: 0
bit: 0
compressed_page: 94e3768
curbufptr: 0
page_cache_hdr[0]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 94d3760
pg_hit_count: 0
page_cache_hdr[1]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 94d4760
pg_hit_count: 0
page_cache_hdr[2]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 94d5760
.... skipped rest of page_cache stuff
page_cache_buf: 94d3760
evict_index: 0
evictions: 0
accesses: 0
cached_reads: 0
valid_pages: 94d3718
... skipped some stuff
<readmem: 804ed6c0, KVADDR, "kernel_config_data", 32768, (ROE), a009250>
WARNING: could not find MAGIC_START!
<readmem: 8066cce8, KVADDR, "cpu_possible_map", 4, (ROE), 8430680>
cpu_possible_map: 0
<readmem: 8066ccec, KVADDR, "cpu_present_map", 4, (ROE), 8430680>
cpu_present_map: 0
<readmem: 8066cce4, KVADDR, "cpu_online_map", 4, (ROE), 8430680>
cpu_online_map: 0
<readmem: 806e09d0, KVADDR, "xtime", 8, (FOE), 8408d38>
<readmem: 80626004, KVADDR, "init_uts_ns", 390, (ROE), 840931c>
base kernel version: 2.6.27
<readmem: 804e8000, KVADDR, "accessible check", 4, (ROE|Q), ff87ce14>
<readmem: 804e8000, KVADDR, "readstring characters", 1499, (ROE|Q), ff87be10>
WARNING: cannot find matching kernel version in
u/boot/vmlinux-2.6.27.21+up-syms file:
verify_namelist:
/proc/version:
Linux version 2.6.27.21+up (root@x) (gcc version 4.1.2 20061115
(prerelease) (Debian 4.1.1-21)) #1 Fri May 15 13:56:19 UTC 2009
utsname version: #1 Fri May 15 13:56:19 UTC 2009
u/boot/vmlinux-2.6.27.21+up-syms:
<readmem: 802577e0, KVADDR, "x86_omit_frame_pointer", 8, (ROE), ff87dfd8>
<readmem: 806e93ec, KVADDR, "high_memory", 4, (FOE), 840b504>
<readmem: 806e93e0, KVADDR, "mem_map", 4, (FOE), 840b50c>
<readmem: 806e940c, KVADDR, "vmlist", 4, (FOE), ff87d974>
<readmem: 8f8028a4, KVADDR, "first vmlist addr", 4, (ROE), ff87d970>
<readmem: 8066cd5c, KVADDR, "totalram_pages", 4, (FOE), 840b514>
.... skipped lot of output
KERNEL: u/boot/vmlinux-2.6.27.21+up-syms
DUMPFILE: kerneldump-20090527-075410 [PARTIAL DUMP]
CPUS: 1
DATE: Wed May 27 10:53:54 2009
<readmem: 8062b8f8, KVADDR, "jiffies_64", 8, (FOE), ff87d9c8>
crash: convert_time: 283566 (453ae)
UPTIME: 00:04:43
<readmem: 806e05e0, KVADDR, "avenrun array", 12, (FOE), ff87d9c0>
LOAD AVERAGE: 0.14, 0.14, 0.06
TASKS: 141
NODENAME: tp3
RELEASE: 2.6.27.21+up
VERSION: #1 Fri May 15 13:56:19 UTC 2009
MACHINE: i686 <readmem: 806bff50, KVADDR, "cpu_khz", 4, (FOE), ff87d9e0>
(2658 Mhz)
MEMORY: 255.5 MB
PANIC: NOTE: GETBUF request > MAX_CACHE_SIZE: 65536
"[ 136.195328] SysRq : Trigger a crashdump"
PID: 3208
COMMAND: "bash"
TASK: 8f1be0c0 [THREAD_INFO: 88d44000]
CPU: 0
<readmem: 80630790, KVADDR, "task_state_array", 4, (ROE), ff87d998>
<readmem: 805b2fd4, KVADDR, "readstring characters", 44, (ROE|Q), ff87c370>
...
------- some output from working SMP standard memory split (but PAE
enabled) crash
mcore_data:
flags: a0 (KDUMP_LOCAL|KDUMP_ELF32)
ndfd: 3
ofp: f7f074c0
header_size: 540
num_pt_load_segments: 4
pt_load_segment[0]:
file_offset: 21c
phys_start: 0
phys_end: a0000
zero_fill: 0
pt_load_segment[1]:
file_offset: a021c
phys_start: 100000
phys_end: 1000000
zero_fill: 0
....
elf_header: 99eb6f8
elf32: 99eb6f8
notes32: 99eb72c
load32: 99eb74c
...
<readmem: c053e700, KVADDR, "kernel_config_data", 32768, (ROE), a5b67f8>
crash: CONFIG_NR_CPUS: 32
crash: CONFIG_HZ: 100
<readmem: c07318c0, KVADDR, "cpu_possible_map", 4, (ROE), 842fea0>
cpu_possible_map: 0 1
<readmem: c06c378c, KVADDR, "cpu_present_map", 4, (ROE), 842fea0>
cpu_present_map: 0 1
<readmem: c06c32a0, KVADDR, "cpu_online_map", 4, (ROE), 842fea0>
cpu_online_map: 0 1
...
PANIC: NOTE: GETBUF request > MAX_CACHE_SIZE: 65536
"[ 31.044652] SysRq : Trigger a crashdump"
PID: 2745
....
---------- end of snips
Thanks in advance,
Tero
15 years, 1 month
Re: crash 4.0-8.9 w/ 2.6.30-rc6
by Dave Anderson
----- "Mike Snitzer" <snitzer(a)redhat.com> wrote:
> On Wed, May 27 2009 at 8:37am -0400,
> Dave Anderson <anderson(a)redhat.com> wrote:
>
> >
> > ----- "Mike Snitzer" <snitzer(a)redhat.com> wrote:
> >
> > > Hi Dave,
> > >
> > > crash is failing with the following when I try to throw a
> 2.6.30-rc6
> > > vmcore at it:
> > >
> > > crash: invalid structure size: x8664_pda
> > > FILE: x86_64.c LINE: 584 FUNCTION: x86_64_cpu_pda_init()
> > >
> > > [/usr/bin/crash] error trace: 449c7f => 4ce815 => 4d00cf =>
> 50936d
> > >
> > > 50936d: SIZE_verify+168
> > > 4d00cf: (undetermined)
> > > 4ce815: x86_64_init+3205
> > > 449c7f: main_loop+152
> > >
> > > I can dig deeper but your help would be very much appreciated.
> > >
> > > Mike
> >
> > The venerable "been-there-since-the-beginning-of-x86_64" x8664_pda
> > data structure no longer exists. It was a per-cpu array of a
> fundamental
> > data structure that things like "current", the per-cpu magic number,
> the
> > cpu number, the current kernel stack pointer, the per-cpu IRQ stack
> pointer,
> > etc. all came from:
> >
> > /* Per processor datastructure. %gs points to it while the kernel
> runs */
> > struct x8664_pda {
> > struct task_struct *pcurrent; /* Current process */
> > unsigned long data_offset; /* Per cpu data offset from
> linker address */
> > unsigned long kernelstack; /* top of kernel stack for
> current */
> > unsigned long oldrsp; /* user rsp for system call */
> > #if DEBUG_STKSZ > EXCEPTION_STKSZ
> > unsigned long debugstack; /* #DB/#BP stack. */
> > #endif
> > int irqcount; /* Irq nesting counter. Starts
> with -1 */
> > int cpunumber; /* Logical CPU number */
> > char *irqstackptr; /* top of irqstack */
> > int nodenumber; /* number of current node */
> > unsigned int __softirq_pending;
> > unsigned int __nmi_count; /* number of NMI on this
> CPUs */
> > int mmu_state;
> > struct mm_struct *active_mm;
> > unsigned apic_timer_irqs;
> > } ____cacheline_aligned_in_smp;
> >
> > There have been upstream rumblings about replacing it with a more efficient
> > per-cpu implementation for some time now, but I haven't studied how the new
> > scheme works yet. It will be a major re-work for the crash utility, so you're
> > pretty much out of luck for now. (Try "gdb vmlinux vmcore" for basic info)
>
> Ah OK. I was just looking to get a stack trace. Unfortunately gdb
> isn't playing nice either:
>
> (gdb) bt
> #0 kstat_irqs_cpu (irq=<value optimized out>, cpu=2) at
> kernel/irq/handle.c:555
> Cannot access memory at address 0xffff88007e5e7d50
Mike,
Try the "--minimal" option that the IBM guys put into 4.0-7.1:
- Implementation of a "--minimal" command line option, which brings
up a crash session that is restricted to the "log", "dis", "rd",
"sym", "eval" and "exit" commands. This option may provide a way to
extract some minimal/quick information from a corrupted or truncated
dumpfile, or in situations where one of the several kernel subsystem
initialization routines, which are not called, would abort the
crash session. (sharyath(a)in.ibm.com, sachinp(a)in.ibm.com)
So just enter this:
$ crash --minimal vmlinux vmcore
And you should at least get the kernel trace info with the "log" command.
Dave
15 years, 1 month
Re: [Crash-utility] Re: [PATCH 0/3] Display local variables & function parameters from stack frames
by Dave Anderson
----- "Sharyathi Nagesh" <sharyath(a)in.ibm.com> wrote:
> Dave
> Excuse me for overlooking this part of the code, I am attaching a fix
> to this, hope this fixes the issue.
That one looks better...
> Dave I have few observations regarding the points you have raised
> mkdumpfile -c stripping the ELF Note for pt_regs:
> mkdumpfile won't be saving much by stripping ELF Notes of pt_regs
> information. It will be ~256 bytes * number of cpus which is not much.
> We will discuss with mkdumpfile developers to check out the possibility
> of retaining this ELF Note information.
>
> Regarding CONFIG_FRAMEPOINTER
> We understand this is disabled so as to release one more
> register,bp, for general purpose operations and this is default.
> Ideally this information should have got saved in dwarf section, so
> theoretically speaking we should be able to unwind the x86/x86_64 dump
> even with out CONFIG_FRAMEPOINTER. But some how the stack unwinding is
> not as direct as it is in ppc64 we are re-looking into this
> implementation.
>
> Regarding Exception Frame on the top of the stack frame
> As we understand if we have the pt_regs of the topmost stack we
> should be able to unwind to the next stack frame, even if top most
> stack frame is an exception frame, atleast in ppc64. We are not sure
> of x86 and x86_64 we can relook into that too.
I understand that with the topmost pt_regs you can then start the
backtrace OK. That's not what I'm referring to.
What I'm talking about is bumping into another exception frame
while unwinding from the topmost pt_regs. Or what happens
when the crash occurs while operating on an alternate kernel
stack.
Just take a simple example -- what happens when you actually enter
"alt-sysrq-c" on an x86_64, generating a keyboard interrupt and therefore
a transition to the per-cpu IRQ stack? By the time crash-kexec()
is called, you're already operating on the IRQ stack. Your
code will work its way back to the top of the per-cpu IRQ stack,
but then what does it do? Or suppose the task takes a page fault,
lays down an exception frame, and then later BUG()'s out while attempting
the handle the fault. Your code will start from the most-recently
occurring exception frame, but will bump into the page fault exception
during the unwind operation. Does you code properly recognize the new
exception frame (and the passage through assembly-language code
when that happens)?
All I'm saying is basing your test results simply on instances where
panic() is called or "echo c ..." was entered is the most trivial type
of kernel crash -- because there's no kernel exception frame
laid down until crash_kexec() gets called. That's a fairly rare
occurance w/respect to typical kernel crashes.
Dave
15 years, 1 month
Re: [PATCH 0/3] Display local variables & function parameters from stack frames
by Dave Anderson
Dave Anderson wrote:
>
> That all being said, here's what I can do for you. I will take your new
> functions that you've added to netdump.c, and their protos in defs.h,
> and apply them to the next crash utility release. Since they are not being
> used by anybody but your module at the moment, it's harmless to add them,
> and then you will not have to patch the crash sources at all in your
> subsequent module patch/posts.
I take that back...
You're first going to have to fix this proposed netdump.c function:
+static void* get_x86_regs_from_elf_notes(struct task_context *tc)
+{
+ Elf32_Nhdr *note;
An x86 kdump can be either a 32-bit or 64-bit ELF vmcore, but the default
is 64-bit. You have defined only one "note" pointer as an Elf32_hdr pointer.
+ size_t len;
+ char *pt_regs;
+ ulong regs_size;
+
+ if((tc->task == tt->panic_task) ||
+ (is_task_active(tc->task))){
+ if (nd->num_prstatus_notes > 1)
+ note = (Elf32_Nhdr *)
+ nd->nt_prstatus_percpu[tc->processor];
+ else
+ note = (Elf64_Nhdr *)nd->nt_prstatus;
And then above you determine first whether there is more than one prstatus
note (i.e., more than one cpu), and then based upon that you're assigning
the "note" pointer to the appropriate location. But for the single-cpu
you make it an Elf64_Nhdr pointer? I realize it's just used as a address
value, but it's just not right. (BTW, compile crash with "make warn"
instead of just "make")
Anyway, you can conceivably have these i386 combinations:
1. single-cpu, a 32-bit ELF header, and 1 prstatus note (netdump or kdump)
2. single-cpu, a 64-bit ELF header, and 1 prstatus note (kdump only)
3. multiple-cpu, a 32-bit ELF header, and 1 prstatus note (netdump only)
4. multiple-cpu, a 32-bit ELF header, and more-than-one prstatus note (kdump)
5. multiple-cpu, a 64-bit ELF header, and more-than-one prstatus note (kdump)
But below you just default to an Elf32_Nhdr structure:
+
+ len = sizeof(Elf32_Nhdr);
+ len = roundup(len + note->n_namesz, 4);
+ pt_regs = (void *)((char *)note + len +
+ MEMBER_OFFSET("elf_prstatus", "pr_reg"));
+ /* NEED TO BE FIXED: Hack to get the proper alignment */
+ pt_regs +=4;
+ }
+ else
+ error(FATAL, "\n cannot determine register set for the task %s"
+ "bailing out\n ", tc->comm);
+ return pt_regs;
You'll first need to determine whether it's a 32-bit or 64-bit header
being used, and then whether to to point to the single prstatus note or
to the correct one in the NR_CPUS-bounded array of prstatus notes.
Dave
15 years, 1 month