7:37 a.m.
----- "Mike Snitzer" <snitzer(a)redhat.com> wrote:
Hi Dave,
crash is failing with the following when I try to throw a 2.6.30-rc6
vmcore at it:
crash: invalid structure size: x8664_pda
FILE: x86_64.c LINE: 584 FUNCTION: x86_64_cpu_pda_init()
[/usr/bin/crash] error trace: 449c7f => 4ce815 => 4d00cf => 50936d
50936d: SIZE_verify+168
4d00cf: (undetermined)
4ce815: x86_64_init+3205
449c7f: main_loop+152
I can dig deeper but your help would be very much appreciated.
Mike
The venerable "been-there-since-the-beginning-of-x86_64" x8664_pda
data structure no longer exists. It was a per-cpu array of a fundamental
data structure that things like "current", the per-cpu magic number, the
cpu number, the current kernel stack pointer, the per-cpu IRQ stack pointer,
etc. all came from:
/* Per processor datastructure. %gs points to it while the kernel runs */
struct x8664_pda {
struct task_struct *pcurrent; /* Current process */
unsigned long data_offset; /* Per cpu data offset from linker address */
unsigned long kernelstack; /* top of kernel stack for current */
unsigned long oldrsp; /* user rsp for system call */
#if DEBUG_STKSZ > EXCEPTION_STKSZ
unsigned long debugstack; /* #DB/#BP stack. */
#endif
int irqcount; /* Irq nesting counter. Starts with -1 */
int cpunumber; /* Logical CPU number */
char *irqstackptr; /* top of irqstack */
int nodenumber; /* number of current node */
unsigned int __softirq_pending;
unsigned int __nmi_count; /* number of NMI on this CPUs */
int mmu_state;
struct mm_struct *active_mm;
unsigned apic_timer_irqs;
} ____cacheline_aligned_in_smp;
There have been upstream rumblings about replacing it with a more efficient
per-cpu implementation for some time now, but I haven't studied how the new
scheme works yet. It will be a major re-work for the crash utility, so you're
pretty much out of luck for now. (Try "gdb vmlinux vmcore" for basic info)
In the meantime, can you give me a copy of your vmcore? (offline -- note that
I'm forwarding this to the crash-utility mailing list). And I'll start working
on it.
Thanks,
Dave
7:52 a.m.
New subject: crash 4.0-8.9 w/ 2.6.30-rc6
On Wed, May 27 2009 at 8:37am -0400,
Dave Anderson <anderson(a)redhat.com> wrote:
----- "Mike Snitzer" <snitzer(a)redhat.com> wrote:
> Hi Dave,
>
> crash is failing with the following when I try to throw a 2.6.30-rc6
> vmcore at it:
>
> crash: invalid structure size: x8664_pda
> FILE: x86_64.c LINE: 584 FUNCTION: x86_64_cpu_pda_init()
>
> [/usr/bin/crash] error trace: 449c7f => 4ce815 => 4d00cf => 50936d
>
> 50936d: SIZE_verify+168
> 4d00cf: (undetermined)
> 4ce815: x86_64_init+3205
> 449c7f: main_loop+152
>
> I can dig deeper but your help would be very much appreciated.
>
> Mike
The venerable "been-there-since-the-beginning-of-x86_64" x8664_pda
data structure no longer exists. It was a per-cpu array of a fundamental
data structure that things like "current", the per-cpu magic number, the
cpu number, the current kernel stack pointer, the per-cpu IRQ stack pointer,
etc. all came from:
/* Per processor datastructure. %gs points to it while the kernel runs */
struct x8664_pda {
struct task_struct *pcurrent; /* Current process */
unsigned long data_offset; /* Per cpu data offset from linker address */
unsigned long kernelstack; /* top of kernel stack for current */
unsigned long oldrsp; /* user rsp for system call */
#if DEBUG_STKSZ > EXCEPTION_STKSZ
unsigned long debugstack; /* #DB/#BP stack. */
#endif
int irqcount; /* Irq nesting counter. Starts with -1 */
int cpunumber; /* Logical CPU number */
char *irqstackptr; /* top of irqstack */
int nodenumber; /* number of current node */
unsigned int __softirq_pending;
unsigned int __nmi_count; /* number of NMI on this CPUs */
int mmu_state;
struct mm_struct *active_mm;
unsigned apic_timer_irqs;
} ____cacheline_aligned_in_smp;
There have been upstream rumblings about replacing it with a more efficient
per-cpu implementation for some time now, but I haven't studied how the new
scheme works yet. It will be a major re-work for the crash utility, so you're
pretty much out of luck for now. (Try "gdb vmlinux vmcore" for basic info)
Ah OK. I was just looking to get a stack trace. Unfortunately gdb
isn't playing nice either:
(gdb) bt
#0 kstat_irqs_cpu (irq=<value optimized out>, cpu=2) at kernel/irq/handle.c:555
Cannot access memory at address 0xffff88007e5e7d50
In the meantime, can you give me a copy of your vmcore? (offline --
note that
I'm forwarding this to the crash-utility mailing list). And I'll start working
on it.
OK, will do.
Mike
8:02 p.m.
New subject: crash 4.0-8.9 w/ 2.6.30-rc6
On Wed, 27 May 2009 08:37:46 -0400 (EDT), Dave Anderson <anderson(a)redhat.com>
wrote:
----- "Mike Snitzer" <snitzer(a)redhat.com> wrote:
> Hi Dave,
>
> crash is failing with the following when I try to throw a 2.6.30-rc6
> vmcore at it:
>
> crash: invalid structure size: x8664_pda
> FILE: x86_64.c LINE: 584 FUNCTION: x86_64_cpu_pda_init()
>
> [/usr/bin/crash] error trace: 449c7f => 4ce815 => 4d00cf => 50936d
>
> 50936d: SIZE_verify+168
> 4d00cf: (undetermined)
> 4ce815: x86_64_init+3205
> 449c7f: main_loop+152
>
> I can dig deeper but your help would be very much appreciated.
>
> Mike
The venerable "been-there-since-the-beginning-of-x86_64" x8664_pda
data structure no longer exists. It was a per-cpu array of a fundamental
data structure that things like "current", the per-cpu magic number, the
cpu number, the current kernel stack pointer, the per-cpu IRQ stack pointer,
etc. all came from:
/* Per processor datastructure. %gs points to it while the kernel runs */
struct x8664_pda {
struct task_struct *pcurrent; /* Current process */
unsigned long data_offset; /* Per cpu data offset from linker address */
unsigned long kernelstack; /* top of kernel stack for current */
unsigned long oldrsp; /* user rsp for system call */
#if DEBUG_STKSZ > EXCEPTION_STKSZ
unsigned long debugstack; /* #DB/#BP stack. */
#endif
int irqcount; /* Irq nesting counter. Starts with -1 */
int cpunumber; /* Logical CPU number */
char *irqstackptr; /* top of irqstack */
int nodenumber; /* number of current node */
unsigned int __softirq_pending;
unsigned int __nmi_count; /* number of NMI on this CPUs */
int mmu_state;
struct mm_struct *active_mm;
unsigned apic_timer_irqs;
} ____cacheline_aligned_in_smp;
There have been upstream rumblings about replacing it with a more efficient
per-cpu implementation for some time now, but I haven't studied how the new
scheme works yet. It will be a major re-work for the crash utility, so you're
pretty much out of luck for now. (Try "gdb vmlinux vmcore" for basic info)
Hi, Dave.
It's a very and very dirty hack, but some of my colleagues say that
a dump file of 2.6.30-rcX can be analyzed by this patch.
I hope it would be some help for you.
Thanks,
Daisuke Nishimura.
Signed-off-by: Daisuke Nishimura <nishimura(a)mxp.nes.nec.co.jp>
---
defs.h | 1 +
x86_64.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----
2 files changed, 74 insertions(+), 5 deletions(-)
diff --git a/defs.h b/defs.h
index e5e3538..32b2e1e 100755
--- a/defs.h
+++ b/defs.h
@@ -3851,6 +3851,7 @@ struct machine_specific {
#define PHYS_BASE (0x80)
#define VM_XEN_RHEL4 (0x100)
#define VMEMMAP (0x200)
+#define KSYMS_PERCPU (0x400)
#define VM_FLAGS (VM_ORIG|VM_2_6_11|VM_XEN|VM_XEN_RHEL4)
diff --git a/x86_64.c b/x86_64.c
index 993827e..4813f96 100755
--- a/x86_64.c
+++ b/x86_64.c
@@ -71,6 +71,7 @@ static int x86_64_is_uvaddr(ulong, struct task_context *);
void x86_64_compiler_warning_stub(void);
static void x86_64_init_kernel_pgd(void);
static void x86_64_cpu_pda_init(void);
+static void x86_64_per_cpu_init(void);
static void x86_64_ist_init(void);
static void x86_64_post_init(void);
static void parse_cmdline_arg(void);
@@ -290,7 +291,10 @@ x86_64_init(int when)
MEMBER_OFFSET_INIT(user_regs_struct_ss,
"user_regs_struct", "ss");
STRUCT_SIZE_INIT(user_regs_struct, "user_regs_struct");
- x86_64_cpu_pda_init();
+ if (STRUCT_EXISTS("x8664_pda"))
+ x86_64_cpu_pda_init();
+ else
+ x86_64_per_cpu_init();
x86_64_ist_init();
if ((machdep->machspec->irqstack = (char *)
malloc(machdep->machspec->stkinfo.isize)) == NULL)
@@ -677,6 +681,51 @@ x86_64_cpu_pda_init(void)
FREEBUF(cpu_pda_buf);
}
+static void
+x86_64_per_cpu_init(void)
+{
+ int i, cpus, cpunumber;
+ ulong istacksize;
+
+ if (!(kt->flags & PER_CPU_OFF))
+ return;
+
+ if (!symbol_exists("per_cpu__cpu_number") ||
!symbol_exists("per_cpu__irq_stack_ptr"))
+ return;
+
+ for (i = cpus = 0; i < NR_CPUS; i++) {
+ readmem(symbol_value("per_cpu__cpu_number") + kt->__per_cpu_offset[i],
+ KVADDR, &cpunumber, sizeof(int),
+ "cpu number (per_cpu)", FAULT_ON_ERROR);
+ if (cpunumber != cpus)
+ break;
+ cpus++;
+
+ readmem(symbol_value("per_cpu__irq_stack_ptr") + kt->__per_cpu_offset[i],
+ KVADDR, &machdep->machspec->stkinfo.ibase[i],
+ sizeof(ulong), "irq stack ptr (per cpu)", FAULT_ON_ERROR);
+ }
+
+ istacksize = 16384; /* 16K */
+ machdep->machspec->stkinfo.isize = istacksize;
+
+ /*
+ * Adjust the kernel top-of-stack values down to their base.
+ */
+ for (i = 0; i < NR_CPUS; i++) {
+ if (machdep->machspec->stkinfo.ibase[i])
+ machdep->machspec->stkinfo.ibase[i] -= (istacksize-64);
+ else
+ break;
+ }
+
+ kt->cpus = cpus;
+ if (kt->cpus > 1)
+ kt->flags |= SMP;
+
+ verify_spinlock();
+}
+
/*
* Gather the ist addresses for each CPU.
*/
@@ -754,7 +803,7 @@ x86_64_ist_init(void)
*/
sp = value_search(ms->stkinfo.ebase[0][0], &offset);
if (!sp || offset || !STREQ(sp->name, "boot_exception_stacks")) {
- if (symbol_value("boot_exception_stacks")) {
+ if (symbol_exists("boot_exception_stacks")) {
error(WARNING,
"cpu 0 first exception stack: %lx\n boot_exception_stacks:
%lx\n\n",
ms->stkinfo.ebase[0][0],
@@ -1706,8 +1755,12 @@ x86_64_verify_symbol(const char *name, ulong value, char type)
{
if (STREQ(name, "_text") || STREQ(name, "_stext"))
machdep->flags |= KSYMS_START;
+ if (STREQ(name, "__per_cpu_start"))
+ machdep->flags |= KSYMS_PERCPU;
+ if (STREQ(name, "__per_cpu_end"))
+ machdep->flags &= ~KSYMS_PERCPU;
- if (!name || !strlen(name) || !(machdep->flags & KSYMS_START))
+ if (!name || !strlen(name) || !(machdep->flags & (KSYMS_START |
KSYMS_PERCPU)))
return FALSE;
return TRUE;
}
@@ -3943,8 +3996,23 @@ x86_64_get_smp_cpus(void)
char *cpu_pda_buf;
ulong level4_pgt, cpu_pda_addr;
- if (!VALID_STRUCT(x8664_pda))
- return 1;
+ if (!VALID_STRUCT(x8664_pda)) {
+ if (!(kt->flags & PER_CPU_OFF))
+ return 1;
+
+ if (!symbol_exists("per_cpu__cpu_number"))
+ return 1;
+
+ for (i = cpus = 0; i < NR_CPUS; i++) {
+ readmem(symbol_value("per_cpu__cpu_number") + kt->__per_cpu_offset[i],
+ KVADDR, &cpunumber, sizeof(int),
+ "cpu number (per_cpu)",FAULT_ON_ERROR);
+ if (cpunumber != cpus)
+ break;
+ cpus++;
+ }
+ return cpus;
+ }
cpu_pda_buf = GETBUF(SIZE(x8664_pda));
In the meantime, can you give me a copy of your vmcore? (offline --
note that
I'm forwarding this to the crash-utility mailing list). And I'll start working
on it.
Thanks,
Dave
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
5661
days inactive
5662
days old
devel@lists.crash-utility.osci.io
2 comments
3 participants
participants (3)
-
Daisuke Nishimura -
Dave Anderson -
Mike Snitzer