12:23 p.m.
This message was bounced due to its size of its attachment;
I've since bumped up the maximum allowable message size:
Re: [Crash-utility] crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Date: Wed, 26 Oct 2005 09:15:47 -0700
From: Badari Pulavarty <pbadari(a)us.ibm.com>
To: <crash-utility(a)redhat.com>
References: 1, 2, 3, 4
On Wed, 2005-10-26 at 11:51 -0400, Dave Anderson wrote:
>
> crash: read error: kernel virtual address: ffff8100050eb084 type:
> "tss_struct ist array"
>
I see that the 2.6.13 kernel defines its init_tss
array like so:
DEFINE_PER_CPU(struct tss_struct, init_tss)
____cacheline_maxaligned_in_smp;
whereas, the earlier 2.6 kernels do it like this:
DECLARE_PER_CPU(struct tss_struct,init_tss);
If this change modifies the way that per-cpu variable addresses
are laid out, then I can't tell you what to do without significant
further investigation. But until proven otherwise, let's presume
that the calculations of the per-cpu data is done the same way.
There are two places where that error message comes from, both
in x86_64_ist_init(), but given that the above per-cpu declarations
are functionally equivalent, there would be the following
kernel symbol in your vmlinux, verifiable like so:
$ nm -Bn vmlinux | grep per_cpu__init_tss
ffffffff80502100 D per_cpu__init_tss
$
If it's not there, crash is hosed, then signficant work needs
to be done to find it. But if the symbol is still intact in
the 2.6.14 kernel, the failure should have come from an incorrect
calculation of the vaddr of the init_tss below:
None of the above stuff changed, so we are fine.
static void
x86_64_ist_init(void)
{
...
} else if (symbol_exists("per_cpu__init_tss")) {
for (c = 0; c < NR_CPUS; c++) {
if ((kt->flags & SMP) && (kt->flags &
PER_CPU_OFF)) {
if (kt->__per_cpu_offset[c] == 0)
break;
vaddr = symbol_value
("per_cpu__init_tss") +
kt->__per_cpu_offset[c];
} else
vaddr = symbol_value
("per_cpu__init_tss");
vaddr += OFFSET(tss_struct_ist);
readmem(vaddr, KVADDR, &ms->stkinfo.ebase
[c][0],
sizeof(ulong) * 7, "tss_struct ist
array",
FAULT_ON_ERROR);
Yes. I realized that the problem is due to messed up
kt->__per_cpu_offset[c] value. These should be offset into the array,
they should be small values. I see huge numbers.
per-cpu offset: 84afdf60
I also realized that this gets set at the lines I touched earlier :(
I can't seem to find out what I screwed up. We are just reading a value
from the kernel structure and setting it.
if (ms->stkinfo.ebase[c][0] == 0)
break;
}
}
I'm also presuming your test kernel is SMP. But I'm wondering
whether
the SMP and PER_CPU_OFF flags are set?
Yes.
The SMP flag should have been pre-set in kernel_init(), but the
PER_CPU_OFF flag gets set in x86_64_cpu_pda_init(), which you
have modified.
You can display the kt->flags contents with a printk x86_64_ist_init
().
If PER_CPU_OFF is not set, then that's probably the issue here.
Can you show your new versions of x86_64_cpu_pda_init() and
x86_64_get_smp_cpus()?
Here are new versions of x64-64 for your review.
Thanks,
Badari
12:49 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Dave Anderson wrote:
This message was bounced due to its size of its attachment;
I've since bumped up the maximum allowable message size:
Re: [Crash-utility] crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Date: Wed, 26 Oct 2005 09:15:47 -0700
From: Badari Pulavarty <pbadari(a)us.ibm.com>
To: <crash-utility(a)redhat.com>
References: 1, 2, 3, 4
On Wed, 2005-10-26 at 11:51 -0400, Dave Anderson wrote:
> >
> > crash: read error: kernel virtual address: ffff8100050eb084 type:
> > "tss_struct ist array"
> >
>
> I see that the 2.6.13 kernel defines its init_tss
> array like so:
>
> DEFINE_PER_CPU(struct tss_struct, init_tss)
> ____cacheline_maxaligned_in_smp;
>
> whereas, the earlier 2.6 kernels do it like this:
>
> DECLARE_PER_CPU(struct tss_struct,init_tss);
>
> If this change modifies the way that per-cpu variable addresses
> are laid out, then I can't tell you what to do without significant
> further investigation. But until proven otherwise, let's presume
> that the calculations of the per-cpu data is done the same way.
>
> There are two places where that error message comes from, both
> in x86_64_ist_init(), but given that the above per-cpu declarations
> are functionally equivalent, there would be the following
> kernel symbol in your vmlinux, verifiable like so:
>
> $ nm -Bn vmlinux | grep per_cpu__init_tss
> ffffffff80502100 D per_cpu__init_tss
> $
>
> If it's not there, crash is hosed, then signficant work needs
> to be done to find it. But if the symbol is still intact in
> the 2.6.14 kernel, the failure should have come from an incorrect
> calculation of the vaddr of the init_tss below:
None of the above stuff changed, so we are fine.
I'm still not convinced that the change from DEFINE_PER_CPU
to DECLARE_PER_CPU is not the culprit -- see below...
> static void
> x86_64_ist_init(void)
> {
> ...
>
> } else if (symbol_exists("per_cpu__init_tss")) {
> for (c = 0; c < NR_CPUS; c++) {
> if ((kt->flags & SMP) && (kt->flags &
> PER_CPU_OFF)) {
> if (kt->__per_cpu_offset[c] == 0)
> break;
> vaddr = symbol_value
> ("per_cpu__init_tss") +
> kt->__per_cpu_offset[c];
> } else
> vaddr = symbol_value
> ("per_cpu__init_tss");
>
> vaddr += OFFSET(tss_struct_ist);
>
> readmem(vaddr, KVADDR, &ms->stkinfo.ebase
> [c][0],
> sizeof(ulong) * 7, "tss_struct ist
> array",
> FAULT_ON_ERROR);
>
Yes. I realized that the problem is due to messed up
kt->__per_cpu_offset[c] value. These should be offset into the array,
they should be small values. I see huge numbers.
per-cpu offset: 84afdf60
Actually they should be rather large huge numbers. On my 4-cpu
x86_64 running RHEL4 (2.6.9-era), the offset values can be seen
below in the __per_cpu_offset[NR_CPUS] array:
crash> help -k
...
kernel_version: 2.6.9
gcc_version: 3.4.4
runq_siblings: 0
__rq_idx[NR_CPUS]: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
__cpu_idx[NR_CPUS]: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
__per_cpu_offset[NR_CPUS]:
0000010081909f60 0000010081911f60 0000010081919f60 0000010081921f60
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
cpu_flags[NR_CPUS]:0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
crash>
The above can be verified by dumping the x8664_pda data structures
with "mach -c", and looking at the data_offset value:
crash> mach -c | grep data_offset
data_offset = 0x10081909f60,
data_offset = 0x10081911f60,
data_offset = 0x10081919f60,
data_offset = 0x10081921f60,
crash>
In order to access any of the kernel's per-cpu data structures, the
value of the relevant "per_cpu__xxxx" symbol values
("per_cpu__init_tss"
in this case), must be added to the per-cpu "data_offset" values found
in the associated x8665_pda structure in order to determine the real
data structure's virtual address.
That's why I'm wondering whether something changed that's associated
with the change from DECLARE_PER_CPU to DEFINE_PER_CPU?
I also realized that this gets set at the lines I touched earlier :(
I can't seem to find out what I screwed up. We are just reading a value
from the kernel structure and setting it.
> if (ms->stkinfo.ebase[c][0] == 0)
> break;
> }
> }
>
> I'm also presuming your test kernel is SMP. But I'm wondering
> whether
> the SMP and PER_CPU_OFF flags are set?
Yes.
> The SMP flag should have been pre-set in kernel_init(), but the
> PER_CPU_OFF flag gets set in x86_64_cpu_pda_init(), which you
> have modified.
>
> You can display the kt->flags contents with a printk x86_64_ist_init
> ().
> If PER_CPU_OFF is not set, then that's probably the issue here.
>
> Can you show your new versions of x86_64_cpu_pda_init() and
> x86_64_get_smp_cpus()?
Here are new versions of x64-64 for your review.
Your changes look OK. That's why I'm wondering whether the
per-cpu scheme has changed? That's kind of scary, because
the per-cpu offset is used in many processor-independent places
in the crash code, not just here for the init_tss structures.
When your code walks through the x8664_pda, dump out some of
the contents as a sanity check, most notably the data_offset
fields.
Thanks,
Dave
1:05 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
On Wed, 2005-10-26 at 13:23 -0400, Dave Anderson wrote:
This message was bounced due to its size of its attachment;
I've since bumped up the maximum allowable message size:
Re: [Crash-utility] crash 4.0-2.8 fails on 2.6.14-rc5
(EM64T)
Date: Wed, 26 Oct 2005 09:15:47 -0700
From: Badari Pulavarty <pbadari(a)us.ibm.com>
To: <crash-utility(a)redhat.com>
References: 1, 2, 3, 4
I already sent you the changed routines (since it didn't let me
send the whole file). I haven't touched any other routines, if
you want to the whole file - I can send it again.
Thanks,
Badari
1:12 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Badari Pulavarty wrote:
On Wed, 2005-10-26 at 13:23 -0400, Dave Anderson wrote:
>
> This message was bounced due to its size of its attachment;
> I've since bumped up the maximum allowable message size:
>
>
> Re: [Crash-utility] crash 4.0-2.8 fails on 2.6.14-rc5
> (EM64T)
> Date: Wed, 26 Oct 2005 09:15:47 -0700
> From: Badari Pulavarty <pbadari(a)us.ibm.com>
> To: <crash-utility(a)redhat.com>
> References: 1, 2, 3, 4
>
I already sent you the changed routines (since it didn't let me
send the whole file). I haven't touched any other routines, if
you want to the whole file - I can send it again.
Right -- but like I said in the last message (did you see my comments
within it?) your changes are exactly what I suggested, and I don't
see why they would cause a problem. But check out my last post
for why I think it may have something to do with the per-cpu data
structure handling in the x86_64 kernel.
Dave
Thanks,
Badari
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
1:41 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Sorry I've generated some unnecesary confusion re: my comments
about the use of DEFINE_PER_CPU and DECLARE_PER_CPU.
That's what I get for trying to multi-task...
Stepping back, the init_tss array is defined in
"arch/x86_64/kernel/init_task.c".
In 2.6.9, it's declared like so:
/*
* per-CPU TSS segments. Threads are completely 'soft' on Linux,
* no more per-task TSS's. The TSS size is kept cacheline-aligned
* so they are allowed to end up in the .data.cacheline_aligned
* section. Since TSS's are completely CPU-local, we want them
* on exact cacheline boundaries, to eliminate cacheline ping-pong.
*/
DEFINE_PER_CPU(struct tss_struct, init_tss) ____cacheline_maxaligned_in_smp;
In 2.6.13, it's slightly different in that it is initialized to INIT_TSS:
/*
* per-CPU TSS segments. Threads are completely 'soft' on Linux,
* no more per-task TSS's. The TSS size is kept cacheline-aligned
* so they are allowed to end up in the .data.cacheline_aligned
* section. Since TSS's are completely CPU-local, we want them
* on exact cacheline boundaries, to eliminate cacheline ping-pong.
*/
DEFINE_PER_CPU(struct tss_struct, init_tss) ____cacheline_maxaligned_in_smp = INIT_TSS;
Both kernels have the same DECLARE_PER_CPU in the
"x86_64/processor.h" header file:
DECLARE_PER_CPU(struct tss_struct,init_tss);
That being the case, and not seeing why the INIT_TSS initialization should
have anything to do with the problem at hand, I am officially stumped at
why the 2.6.14 kernel shows the problem with your patch.
Dave
2:08 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
On Wed, 2005-10-26 at 14:41 -0400, Dave Anderson wrote:
Sorry I've generated some unnecesary confusion re: my comments
about the use of DEFINE_PER_CPU and DECLARE_PER_CPU.
That's what I get for trying to multi-task...
Stepping back, the init_tss array is defined in
"arch/x86_64/kernel/init_task.c".
In 2.6.9, it's declared like so:
/*
* per-CPU TSS segments. Threads are completely 'soft' on Linux,
* no more per-task TSS's. The TSS size is kept cacheline-aligned
* so they are allowed to end up in the .data.cacheline_aligned
* section. Since TSS's are completely CPU-local, we want them
* on exact cacheline boundaries, to eliminate cacheline ping-pong.
*/
DEFINE_PER_CPU(struct tss_struct, init_tss) ____cacheline_maxaligned_in_smp;
In 2.6.13, it's slightly different in that it is initialized to INIT_TSS:
/*
* per-CPU TSS segments. Threads are completely 'soft' on Linux,
* no more per-task TSS's. The TSS size is kept cacheline-aligned
* so they are allowed to end up in the .data.cacheline_aligned
* section. Since TSS's are completely CPU-local, we want them
* on exact cacheline boundaries, to eliminate cacheline ping-pong.
*/
DEFINE_PER_CPU(struct tss_struct, init_tss) ____cacheline_maxaligned_in_smp = INIT_TSS;
Both kernels have the same DECLARE_PER_CPU in the
"x86_64/processor.h" header file:
DECLARE_PER_CPU(struct tss_struct,init_tss);
That being the case, and not seeing why the INIT_TSS initialization should
have anything to do with the problem at hand, I am officially stumped at
why the 2.6.14 kernel shows the problem with your patch.
Okay, I thought so too. I will take a closer look at it and let you
know what I find. I am tempted to go back to 2.6.10 and see if
crash works. Do you know the last known good kernel release for crash
to work ?
Thanks,
Badari
3:27 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Badari Pulavarty wrote:
On Wed, 2005-10-26 at 14:41 -0400, Dave Anderson wrote:
> Sorry I've generated some unnecesary confusion re: my comments
> about the use of DEFINE_PER_CPU and DECLARE_PER_CPU.
> That's what I get for trying to multi-task...
>
> Stepping back, the init_tss array is defined in
"arch/x86_64/kernel/init_task.c".
>
> In 2.6.9, it's declared like so:
>
> /*
> * per-CPU TSS segments. Threads are completely 'soft' on Linux,
> * no more per-task TSS's. The TSS size is kept cacheline-aligned
> * so they are allowed to end up in the .data.cacheline_aligned
> * section. Since TSS's are completely CPU-local, we want them
> * on exact cacheline boundaries, to eliminate cacheline ping-pong.
> */
> DEFINE_PER_CPU(struct tss_struct, init_tss) ____cacheline_maxaligned_in_smp;
>
> In 2.6.13, it's slightly different in that it is initialized to INIT_TSS:
>
> /*
> * per-CPU TSS segments. Threads are completely 'soft' on Linux,
> * no more per-task TSS's. The TSS size is kept cacheline-aligned
> * so they are allowed to end up in the .data.cacheline_aligned
> * section. Since TSS's are completely CPU-local, we want them
> * on exact cacheline boundaries, to eliminate cacheline ping-pong.
> */
> DEFINE_PER_CPU(struct tss_struct, init_tss) ____cacheline_maxaligned_in_smp =
INIT_TSS;
>
> Both kernels have the same DECLARE_PER_CPU in the
> "x86_64/processor.h" header file:
>
> DECLARE_PER_CPU(struct tss_struct,init_tss);
>
> That being the case, and not seeing why the INIT_TSS initialization should
> have anything to do with the problem at hand, I am officially stumped at
> why the 2.6.14 kernel shows the problem with your patch.
Okay, I thought so too. I will take a closer look at it and let you
know what I find. I am tempted to go back to 2.6.10 and see if
crash works. Do you know the last known good kernel release for crash
to work ?
Sorry -- for x86_64, I can't say that I do know the last version
that worked. Maybe somebody else on the list that uses other
than Red Hat RHEL4 kernels does?
Dave
4:19 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
On Wed, 2005-10-26 at 16:27 -0400, Dave Anderson wrote:
Badari Pulavarty wrote:
> On Wed, 2005-10-26 at 14:41 -0400, Dave Anderson wrote:
> > Sorry I've generated some unnecesary confusion re: my comments
> > about the use of DEFINE_PER_CPU and DECLARE_PER_CPU.
> > That's what I get for trying to multi-task...
> >
> > Stepping back, the init_tss array is defined in
"arch/x86_64/kernel/init_task.c".
> >
> > In 2.6.9, it's declared like so:
> >
> > /*
> > * per-CPU TSS segments. Threads are completely 'soft' on Linux,
> > * no more per-task TSS's. The TSS size is kept cacheline-aligned
> > * so they are allowed to end up in the .data.cacheline_aligned
> > * section. Since TSS's are completely CPU-local, we want them
> > * on exact cacheline boundaries, to eliminate cacheline ping-pong.
> > */
> > DEFINE_PER_CPU(struct tss_struct, init_tss) ____cacheline_maxaligned_in_smp;
> >
> > In 2.6.13, it's slightly different in that it is initialized to INIT_TSS:
> >
> > /*
> > * per-CPU TSS segments. Threads are completely 'soft' on Linux,
> > * no more per-task TSS's. The TSS size is kept cacheline-aligned
> > * so they are allowed to end up in the .data.cacheline_aligned
> > * section. Since TSS's are completely CPU-local, we want them
> > * on exact cacheline boundaries, to eliminate cacheline ping-pong.
> > */
> > DEFINE_PER_CPU(struct tss_struct, init_tss) ____cacheline_maxaligned_in_smp =
INIT_TSS;
> >
> > Both kernels have the same DECLARE_PER_CPU in the
> > "x86_64/processor.h" header file:
> >
> > DECLARE_PER_CPU(struct tss_struct,init_tss);
> >
> > That being the case, and not seeing why the INIT_TSS initialization should
> > have anything to do with the problem at hand, I am officially stumped at
> > why the 2.6.14 kernel shows the problem with your patch.
>
> Okay, I thought so too. I will take a closer look at it and let you
> know what I find. I am tempted to go back to 2.6.10 and see if
> crash works. Do you know the last known good kernel release for crash
> to work ?
>
Sorry -- for x86_64, I can't say that I do know the last version
that worked. Maybe somebody else on the list that uses other
than Red Hat RHEL4 kernels does?
Dave
Dave,
I tried 2.6.10 and crash worked fine there. Here is the what I found
interesting. On 2.6.10 the values seem reasonable, but on 2.6.14 they
have huge values.
2.6.10:
cpunum: 0 data_offset 10084b80f60
cpunum: 1 data_offset 10084b88f60
2.6.14-rc5:
cpunum: 0 data_offset ffff810084af5f60
cpunum: 1 data_offset ffff810084afdf60
I got curious on the top "0xffff8" part an trimmed them.
(basically I did data_offset & 0x00000fffffffffff).
Now I run into next problem :( I am missing something basic.
crash: read error: kernel virtual address: ffff81000000fa90 type:
"pglist_data node_next"
Thanks,
Badari
4:48 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Badari Pulavarty wrote:
On Wed, 2005-10-26 at 16:27 -0400, Dave Anderson wrote:
> Badari Pulavarty wrote:
>
> > On Wed, 2005-10-26 at 14:41 -0400, Dave Anderson wrote:
> > > Sorry I've generated some unnecesary confusion re: my comments
> > > about the use of DEFINE_PER_CPU and DECLARE_PER_CPU.
> > > That's what I get for trying to multi-task...
> > >
> > > Stepping back, the init_tss array is defined in
"arch/x86_64/kernel/init_task.c".
> > >
> > > In 2.6.9, it's declared like so:
> > >
> > > /*
> > > * per-CPU TSS segments. Threads are completely 'soft' on Linux,
> > > * no more per-task TSS's. The TSS size is kept cacheline-aligned
> > > * so they are allowed to end up in the .data.cacheline_aligned
> > > * section. Since TSS's are completely CPU-local, we want them
> > > * on exact cacheline boundaries, to eliminate cacheline ping-pong.
> > > */
> > > DEFINE_PER_CPU(struct tss_struct, init_tss)
____cacheline_maxaligned_in_smp;
> > >
> > > In 2.6.13, it's slightly different in that it is initialized to
INIT_TSS:
> > >
> > > /*
> > > * per-CPU TSS segments. Threads are completely 'soft' on Linux,
> > > * no more per-task TSS's. The TSS size is kept cacheline-aligned
> > > * so they are allowed to end up in the .data.cacheline_aligned
> > > * section. Since TSS's are completely CPU-local, we want them
> > > * on exact cacheline boundaries, to eliminate cacheline ping-pong.
> > > */
> > > DEFINE_PER_CPU(struct tss_struct, init_tss)
____cacheline_maxaligned_in_smp = INIT_TSS;
> > >
> > > Both kernels have the same DECLARE_PER_CPU in the
> > > "x86_64/processor.h" header file:
> > >
> > > DECLARE_PER_CPU(struct tss_struct,init_tss);
> > >
> > > That being the case, and not seeing why the INIT_TSS initialization
should
> > > have anything to do with the problem at hand, I am officially stumped at
> > > why the 2.6.14 kernel shows the problem with your patch.
> >
> > Okay, I thought so too. I will take a closer look at it and let you
> > know what I find. I am tempted to go back to 2.6.10 and see if
> > crash works. Do you know the last known good kernel release for crash
> > to work ?
> >
>
> Sorry -- for x86_64, I can't say that I do know the last version
> that worked. Maybe somebody else on the list that uses other
> than Red Hat RHEL4 kernels does?
>
> Dave
>
Dave,
I tried 2.6.10 and crash worked fine there. Here is the what I found
interesting. On 2.6.10 the values seem reasonable, but on 2.6.14 they
have huge values.
2.6.10:
cpunum: 0 data_offset 10084b80f60
cpunum: 1 data_offset 10084b88f60
2.6.14-rc5:
cpunum: 0 data_offset ffff810084af5f60
cpunum: 1 data_offset ffff810084afdf60
I got curious on the top "0xffff8" part an trimmed them.
(basically I did data_offset & 0x00000fffffffffff).
Well that certainly needs further explanation...
Now I run into next problem :( I am missing something basic.
crash: read error: kernel virtual address: ffff81000000fa90 type:
"pglist_data node_next"
That's probably coming from node_table_init(). Could the pglist_data
list now be using per-cpu data structures? But again, I don't understand
the significance of the ffff8 at the top of the address.
Dave
Thanks,
Badari
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
6:22 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
On Wed, 2005-10-26 at 17:48 -0400, Dave Anderson wrote:
Badari Pulavarty wrote:
> On Wed, 2005-10-26 at 16:27 -0400, Dave Anderson wrote:
> > Badari Pulavarty wrote:
> >
> > > On Wed, 2005-10-26 at 14:41 -0400, Dave Anderson wrote:
> > > > Sorry I've generated some unnecesary confusion re: my comments
> > > > about the use of DEFINE_PER_CPU and DECLARE_PER_CPU.
> > > > That's what I get for trying to multi-task...
> > > >
> > > > Stepping back, the init_tss array is defined in
"arch/x86_64/kernel/init_task.c".
> > > >
> > > > In 2.6.9, it's declared like so:
> > > >
> > > > /*
> > > > * per-CPU TSS segments. Threads are completely 'soft' on
Linux,
> > > > * no more per-task TSS's. The TSS size is kept
cacheline-aligned
> > > > * so they are allowed to end up in the .data.cacheline_aligned
> > > > * section. Since TSS's are completely CPU-local, we want them
> > > > * on exact cacheline boundaries, to eliminate cacheline ping-pong.
> > > > */
> > > > DEFINE_PER_CPU(struct tss_struct, init_tss)
____cacheline_maxaligned_in_smp;
> > > >
> > > > In 2.6.13, it's slightly different in that it is initialized to
INIT_TSS:
> > > >
> > > > /*
> > > > * per-CPU TSS segments. Threads are completely 'soft' on
Linux,
> > > > * no more per-task TSS's. The TSS size is kept
cacheline-aligned
> > > > * so they are allowed to end up in the .data.cacheline_aligned
> > > > * section. Since TSS's are completely CPU-local, we want them
> > > > * on exact cacheline boundaries, to eliminate cacheline ping-pong.
> > > > */
> > > > DEFINE_PER_CPU(struct tss_struct, init_tss)
____cacheline_maxaligned_in_smp = INIT_TSS;
> > > >
> > > > Both kernels have the same DECLARE_PER_CPU in the
> > > > "x86_64/processor.h" header file:
> > > >
> > > > DECLARE_PER_CPU(struct tss_struct,init_tss);
> > > >
> > > > That being the case, and not seeing why the INIT_TSS initialization
should
> > > > have anything to do with the problem at hand, I am officially stumped
at
> > > > why the 2.6.14 kernel shows the problem with your patch.
> > >
> > > Okay, I thought so too. I will take a closer look at it and let you
> > > know what I find. I am tempted to go back to 2.6.10 and see if
> > > crash works. Do you know the last known good kernel release for crash
> > > to work ?
> > >
> >
> > Sorry -- for x86_64, I can't say that I do know the last version
> > that worked. Maybe somebody else on the list that uses other
> > than Red Hat RHEL4 kernels does?
> >
> > Dave
> >
>
> Dave,
>
> I tried 2.6.10 and crash worked fine there. Here is the what I found
> interesting. On 2.6.10 the values seem reasonable, but on 2.6.14 they
> have huge values.
>
> 2.6.10:
> cpunum: 0 data_offset 10084b80f60
> cpunum: 1 data_offset 10084b88f60
>
> 2.6.14-rc5:
>
> cpunum: 0 data_offset ffff810084af5f60
> cpunum: 1 data_offset ffff810084afdf60
>
> I got curious on the top "0xffff8" part an trimmed them.
> (basically I did data_offset & 0x00000fffffffffff).
Well that certainly needs further explanation...
>
> Now I run into next problem :( I am missing something basic.
>
> crash: read error: kernel virtual address: ffff81000000fa90 type:
> "pglist_data node_next"
>
That's probably coming from node_table_init(). Could the pglist_data
list now be using per-cpu data structures? But again, I don't understand
the significance of the ffff8 at the top of the address.
I don't know either. I did some more digging around adding printk()
in 2.6.10, 2.6.14-rc5 kernel where data_offset is getting set.
Interestingly, allocmem() returns different types of address in 2.6.10 &
2.6.14-rc5 (which is causing this huge numbers).
arch/x86_64/kernel/setup.c: setup_per_cpu_areas()
for_each_cpu_mask (i, cpu_possible_map) {
char *ptr;
if (!NODE_DATA(cpu_to_node(i))) {
printk("cpu with no node %d, num_online_nodes %d
\n",
i, num_online_nodes());
ptr = alloc_bootmem(size);
} else {
ptr = alloc_bootmem_node(NODE_DATA(cpu_to_node
(i)), size);
}
if (!ptr)
panic("Cannot allocate cpu data for CPU %d\n",
i);
cpu_pda[i].data_offset = ptr - __per_cpu_start;
printk("i %d ptr %p cpustart %p offset %lx\n", i, ptr, __per_cpu_start,
cpu_pda[i].data_offset);
memcpy(ptr, __per_cpu_start, __per_cpu_end -
__per_cpu_start);
}
Here is the output:
2.6.14-rc5
i 0 ptr ffff8100050eb000 cpustart ffffffff805f50a0 offset
ffff810084af5f60
i 1 ptr ffff8100050f3000 cpustart ffffffff805f50a0 offset
ffff810084afdf60
2.6.10
i 0 ptr 0000010005100000 cpustart ffffffff8057f0a0 offset 10084b80f60
i 1 ptr 0000010005108000 cpustart ffffffff8057f0a0 offset 10084b88f60
i 2 ptr 0000010005110000 cpustart ffffffff8057f0a0 offset 10084b90f60
i 3 ptr 0000010005118000 cpustart ffffffff8057f0a0 offset 10084b98f60
i 4 ptr 0000010005120000 cpustart ffffffff8057f0a0 offset 10084ba0f60
i 5 ptr 0000010005128000 cpustart ffffffff8057f0a0 offset 10084ba8f60
i 6 ptr 0000010005130000 cpustart ffffffff8057f0a0 offset 10084bb0f60
i 7 ptr 0000010005138000 cpustart ffffffff8057f0a0 offset 10084bb8f60
Don't know enough about x86-64 stuff. May be I can ask Andi Kleen
about this change ?
Thanks,
Badari
8:28 a.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Badari Pulavarty wrote:
> > Now I run into next problem :( I am missing something basic.
> >
> > crash: read error: kernel virtual address: ffff81000000fa90 type:
> > "pglist_data node_next"
> >
>
> That's probably coming from node_table_init(). Could the pglist_data
> list now be using per-cpu data structures? But again, I don't understand
> the significance of the ffff8 at the top of the address.
I don't know either. I did some more digging around adding printk()
in 2.6.10, 2.6.14-rc5 kernel where data_offset is getting set.
Interestingly, allocmem() returns different types of address in 2.6.10 &
2.6.14-rc5 (which is causing this huge numbers).
arch/x86_64/kernel/setup.c: setup_per_cpu_areas()
for_each_cpu_mask (i, cpu_possible_map) {
char *ptr;
if (!NODE_DATA(cpu_to_node(i))) {
printk("cpu with no node %d, num_online_nodes %d
\n",
i, num_online_nodes());
ptr = alloc_bootmem(size);
} else {
ptr = alloc_bootmem_node(NODE_DATA(cpu_to_node
(i)), size);
}
if (!ptr)
panic("Cannot allocate cpu data for CPU %d\n",
i);
cpu_pda[i].data_offset = ptr - __per_cpu_start;
printk("i %d ptr %p cpustart %p offset %lx\n", i, ptr, __per_cpu_start,
cpu_pda[i].data_offset);
memcpy(ptr, __per_cpu_start, __per_cpu_end -
__per_cpu_start);
}
Here is the output:
2.6.14-rc5
i 0 ptr ffff8100050eb000 cpustart ffffffff805f50a0 offset
ffff810084af5f60
i 1 ptr ffff8100050f3000 cpustart ffffffff805f50a0 offset
ffff810084afdf60
2.6.10
i 0 ptr 0000010005100000 cpustart ffffffff8057f0a0 offset 10084b80f60
i 1 ptr 0000010005108000 cpustart ffffffff8057f0a0 offset 10084b88f60
i 2 ptr 0000010005110000 cpustart ffffffff8057f0a0 offset 10084b90f60
i 3 ptr 0000010005118000 cpustart ffffffff8057f0a0 offset 10084b98f60
i 4 ptr 0000010005120000 cpustart ffffffff8057f0a0 offset 10084ba0f60
i 5 ptr 0000010005128000 cpustart ffffffff8057f0a0 offset 10084ba8f60
i 6 ptr 0000010005130000 cpustart ffffffff8057f0a0 offset 10084bb0f60
i 7 ptr 0000010005138000 cpustart ffffffff8057f0a0 offset 10084bb8f60
Don't know enough about x86-64 stuff. May be I can ask Andi Kleen
about this change ?
That debug output certainly seems to pinpoint the issue at hand, doesn't it?
Very interesting...
What's strange is that the usage of the cpu_pda[i].data_offset by the
per_cpu() macro in "include/asm-x86_64/percpu.h" is unchanged.
It's probably something very simple going on here, but I don't have
any more ideas at this point.
Dave
11:51 a.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
That debug output certainly seems to pinpoint the issue at hand,
doesn't it?
Very interesting...
What's strange is that the usage of the cpu_pda[i].data_offset by the
per_cpu() macro in "include/asm-x86_64/percpu.h" is unchanged.
It's probably something very simple going on here, but I don't have
any more ideas at this point.
This is the reply I got from Andi Kleen..
-------- Forwarded Message --------
From: Andi Kleen <ak(a)suse.de>
To: Badari Pulavarty <pbadari(a)us.ibm.com>
Subject: Re: cpu_pda->data_offset changed recently ?
Date: Thu, 27 Oct 2005 16:58:54 +0200
On Thursday 27 October 2005 16:53, Badari Pulavarty wrote:
Hi Andi,
I am trying to fix "crash" utility to make it work on 2.6.14-rc5.
(Its running fine on 2.6.10). It looks like crash utility reads
and uses cpu_pda->data_offset values. It looks like there is a
change between 2.6.10 & 2.6.14-rc5 which is causing "data_offset"
to be huge values - which is causing "crash" to break.
I added printk() to find out why ? As you can see from following
what changed - Is this expected ? Please let me know.
bootmem used to allocate from the end of the direct mapping on NUMA
systems. Now it starts at the beginning, often before the kernel .text.
This means it is negative. Perfectly legitimate. crash just has to
handle it.
-Andi
12:17 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Badari Pulavarty wrote:
> That debug output certainly seems to pinpoint the issue at hand,
doesn't it?
> Very interesting...
>
> What's strange is that the usage of the cpu_pda[i].data_offset by the
> per_cpu() macro in "include/asm-x86_64/percpu.h" is unchanged.
>
> It's probably something very simple going on here, but I don't have
> any more ideas at this point.
This is the reply I got from Andi Kleen..
-------- Forwarded Message --------
From: Andi Kleen <ak(a)suse.de>
To: Badari Pulavarty <pbadari(a)us.ibm.com>
Subject: Re: cpu_pda->data_offset changed recently ?
Date: Thu, 27 Oct 2005 16:58:54 +0200
On Thursday 27 October 2005 16:53, Badari Pulavarty wrote:
> Hi Andi,
>
> I am trying to fix "crash" utility to make it work on 2.6.14-rc5.
> (Its running fine on 2.6.10). It looks like crash utility reads
> and uses cpu_pda->data_offset values. It looks like there is a
> change between 2.6.10 & 2.6.14-rc5 which is causing "data_offset"
> to be huge values - which is causing "crash" to break.
>
> I added printk() to find out why ? As you can see from following
> what changed - Is this expected ? Please let me know.
bootmem used to allocate from the end of the direct mapping on NUMA
systems. Now it starts at the beginning, often before the kernel .text.
This means it is negative. Perfectly legitimate. crash just has to
handle it.
-Andi
--
That's what I thought it looked like, although the x8664_pda.data_offset
field is an "unsigned long". Anyway, if you take any of the per_cpu__xxx
symbols from the 2.6.14 kernel, subtract a cpu data_offset, does it come up
with a legitimate virtual address?
Dave
12:28 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
On Thu, 2005-10-27 at 13:17 -0400, Dave Anderson wrote:
Badari Pulavarty wrote:
> > That debug output certainly seems to pinpoint the issue at hand,
> doesn't it?
> > Very interesting...
> >
> > What's strange is that the usage of the cpu_pda[i].data_offset by
> the
> > per_cpu() macro in "include/asm-x86_64/percpu.h" is unchanged.
> >
> > It's probably something very simple going on here, but I don't
> have
> > any more ideas at this point.
>
> This is the reply I got from Andi Kleen..
>
> -------- Forwarded Message --------
> From: Andi Kleen <ak(a)suse.de>
> To: Badari Pulavarty <pbadari(a)us.ibm.com>
> Subject: Re: cpu_pda->data_offset changed recently ?
> Date: Thu, 27 Oct 2005 16:58:54 +0200
> On Thursday 27 October 2005 16:53, Badari Pulavarty wrote:
> > Hi Andi,
> >
> > I am trying to fix "crash" utility to make it work on 2.6.14-rc5.
> > (Its running fine on 2.6.10). It looks like crash utility reads
> > and uses cpu_pda->data_offset values. It looks like there is a
> > change between 2.6.10 & 2.6.14-rc5 which is causing "data_offset"
> > to be huge values - which is causing "crash" to break.
> >
> > I added printk() to find out why ? As you can see from following
> > what changed - Is this expected ? Please let me know.
>
> bootmem used to allocate from the end of the direct mapping on NUMA
> systems. Now it starts at the beginning, often before the
> kernel .text.
> This means it is negative. Perfectly legitimate. crash just has to
> handle it.
>
> -Andi
>
> --
>
That's what I thought it looked like, although the
x8664_pda.data_offset
field is an "unsigned long". Anyway, if you take any of the
per_cpu__xxx
symbols from the 2.6.14 kernel, subtract a cpu data_offset, does it
come up
with a legitimate virtual address?
Unfortunately, I don't know x86-64 kernel virtual address space
well enough to answer your question.
My understanding is x86-64 kernel addresses look something like:
addr: ffffffff80101000
But now (2.6.14-rc5) I do see address like:
pgdat: 0xffff81000000e000
which are causing read problems.
crash: read error: kernel virtual address: ffff81000000fa90 type:
"pglist_data node_next"
I am not sure what these address are and if they are valid.
Is there a way to verify these addresses, through gdb or /dev/kmem
or something like that ?
Thanks,
Badari
1:16 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Badari Pulavarty wrote:
On Thu, 2005-10-27 at 13:17 -0400, Dave Anderson wrote:
> Badari Pulavarty wrote:
> > > That debug output certainly seems to pinpoint the issue at hand,
> > doesn't it?
> > > Very interesting...
> > >
> > > What's strange is that the usage of the cpu_pda[i].data_offset by
> > the
> > > per_cpu() macro in "include/asm-x86_64/percpu.h" is unchanged.
> > >
> > > It's probably something very simple going on here, but I don't
> > have
> > > any more ideas at this point.
> >
> > This is the reply I got from Andi Kleen..
> >
> > -------- Forwarded Message --------
> > From: Andi Kleen <ak(a)suse.de>
> > To: Badari Pulavarty <pbadari(a)us.ibm.com>
> > Subject: Re: cpu_pda->data_offset changed recently ?
> > Date: Thu, 27 Oct 2005 16:58:54 +0200
> > On Thursday 27 October 2005 16:53, Badari Pulavarty wrote:
> > > Hi Andi,
> > >
> > > I am trying to fix "crash" utility to make it work on
2.6.14-rc5.
> > > (Its running fine on 2.6.10). It looks like crash utility reads
> > > and uses cpu_pda->data_offset values. It looks like there is a
> > > change between 2.6.10 & 2.6.14-rc5 which is causing
"data_offset"
> > > to be huge values - which is causing "crash" to break.
> > >
> > > I added printk() to find out why ? As you can see from following
> > > what changed - Is this expected ? Please let me know.
> >
> > bootmem used to allocate from the end of the direct mapping on NUMA
> > systems. Now it starts at the beginning, often before the
> > kernel .text.
> > This means it is negative. Perfectly legitimate. crash just has to
> > handle it.
> >
> > -Andi
> >
> > --
> >
> That's what I thought it looked like, although the
> x8664_pda.data_offset
> field is an "unsigned long". Anyway, if you take any of the
> per_cpu__xxx
> symbols from the 2.6.14 kernel, subtract a cpu data_offset, does it
> come up
> with a legitimate virtual address?
Unfortunately, I don't know x86-64 kernel virtual address space
well enough to answer your question.
My understanding is x86-64 kernel addresses look something like:
addr: ffffffff80101000
But now (2.6.14-rc5) I do see address like:
pgdat: 0xffff81000000e000
which are causing read problems.
crash: read error: kernel virtual address: ffff81000000fa90 type:
"pglist_data node_next"
I am not sure what these address are and if they are valid.
Is there a way to verify these addresses, through gdb or /dev/kmem
or something like that ?
Thanks,
Badari
Here is bottom line we need to understand to fix
the problem.
2.6.10:
pgdat: 0x1000000e000
2.6.14-rc5:
pgdat: 0xffff81000000e000
Exactly.
On a 2.6.9 kernel, if you do an nm -Bn on the vmlinux file, you'll first
see a bunch of "A" type absolute symbols, followed by the text
symbols, then readonly data, data, and so on. Eventually you'll
bump into the per-cpu symbols:
$ nm -Bn vmlinux
0000000000088861 A __crc_dev_mc_delete
000000000014bfd1 A __crc_smp_call_function
00000000002de2e0 A __crc___skb_linearize
0000000000442f14 A __crc_tty_register_device
000000000060e766 A __crc_tty_termios_baud_rate
0000000000712c54 A __crc_remove_inode_hash
00000000007f8e0b A __crc_xfrm_policy_alloc
0000000000801678 A __crc_flush_scheduled_work
0000000000a64d75 A __crc_neigh_changeaddr
... <snip>
00000000ffdf0b3d A __crc_usb_driver_release_interface
00000000ffe031fc A __crc_udp_proc_unregister
00000000ffead192 A __crc_cdrom_number_of_slots
00000000fff9536b A __crc_sock_no_recvmsg
00000000fffb8df8 A __crc_device_unregister
ffffffff80100000 t startup_32
ffffffff80100000 A _text
ffffffff80100081 t reach_compatibility_mode
ffffffff8010008e t second
ffffffff80100100 t reach_long64
ffffffff8010013d T initial_code
ffffffff80100145 T init_rsp
ffffffff80100150 T no_long_mode
ffffffff80100f00 T pGDT32
ffffffff80100f10 t ljumpvector
ffffffff80100f18 T stext
ffffffff80100f18 T _stext
ffffffff80101000 T init_level4_pgt
ffffffff80102000 T level3_ident_pgt
... <snip>
ffffffff80502100 D per_cpu__init_tss
ffffffff80502200 d per_cpu__prof_old_multiplier
ffffffff80502204 d per_cpu__prof_multiplier
ffffffff80502208 d per_cpu__prof_counter
ffffffff80502220 D per_cpu__mmu_gathers
ffffffff80503280 D per_cpu__kstat
ffffffff80503680 d per_cpu__runqueues
ffffffff805048e0 d per_cpu__cpu_domains
ffffffff80504940 d per_cpu__phys_domains
ffffffff805049a0 d per_cpu__node_domains
ffffffff805049f8 D per_cpu__process_counts
ffffffff80504a00 d per_cpu__tasklet_hi_vec
ffffffff80504a08 d per_cpu__tasklet_vec
ffffffff80504a10 d per_cpu__ksoftirqd
ffffffff80504a80 d per_cpu__tvec_bases
ffffffff80506b00 D per_cpu__rcu_bh_data
ffffffff80506b60 D per_cpu__rcu_data
ffffffff80506bc0 d per_cpu__rcu_tasklet
...
So for any data that was specifically created per-cpu,
the symbol above is the starting point, but to get to
the per-cpu structure, the offset value from the
cpu_data.data_offset needs to be applied.
What I don't understand is where the 0xffff810000000000
addresses come into play. Are you seeing them as actual
symbols?
Dave
1:27 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
On Thu, 2005-10-27 at 14:16 -0400, Dave Anderson wrote:
Badari Pulavarty wrote:
> On Thu, 2005-10-27 at 13:17 -0400, Dave Anderson wrote:
> > Badari Pulavarty wrote:
> > > > That debug output certainly seems to pinpoint the issue at
> hand,
> > > doesn't it?
> > > > Very interesting...
> > > >
> > > > What's strange is that the usage of the cpu_pda[i].data_offset
> by
> > > the
> > > > per_cpu() macro in "include/asm-x86_64/percpu.h" is
> unchanged.
> > > >
> > > > It's probably something very simple going on here, but I
> don't
> > > have
> > > > any more ideas at this point.
> > >
> > > This is the reply I got from Andi Kleen..
> > >
> > > -------- Forwarded Message --------
> > > From: Andi Kleen <ak(a)suse.de>
> > > To: Badari Pulavarty <pbadari(a)us.ibm.com>
> > > Subject: Re: cpu_pda->data_offset changed recently ?
> > > Date: Thu, 27 Oct 2005 16:58:54 +0200
> > > On Thursday 27 October 2005 16:53, Badari Pulavarty wrote:
> > > > Hi Andi,
> > > >
> > > > I am trying to fix "crash" utility to make it work on
2.6.14-
> rc5.
> > > > (Its running fine on 2.6.10). It looks like crash utility
> reads
> > > > and uses cpu_pda->data_offset values. It looks like there is
> a
> > > > change between 2.6.10 & 2.6.14-rc5 which is causing
> "data_offset"
> > > > to be huge values - which is causing "crash" to break.
> > > >
> > > > I added printk() to find out why ? As you can see from
> following
> > > > what changed - Is this expected ? Please let me know.
> > >
> > > bootmem used to allocate from the end of the direct mapping on
> NUMA
> > > systems. Now it starts at the beginning, often before the
> > > kernel .text.
> > > This means it is negative. Perfectly legitimate. crash just has
> to
> > > handle it.
> > >
> > > -Andi
> > >
> > > --
> > >
> > That's what I thought it looked like, although the
> > x8664_pda.data_offset
> > field is an "unsigned long". Anyway, if you take any of the
> > per_cpu__xxx
> > symbols from the 2.6.14 kernel, subtract a cpu data_offset, does
> it
> > come up
> > with a legitimate virtual address?
>
> Unfortunately, I don't know x86-64 kernel virtual address space
> well enough to answer your question.
>
> My understanding is x86-64 kernel addresses look something like:
>
> addr: ffffffff80101000
>
> But now (2.6.14-rc5) I do see address like:
>
> pgdat: 0xffff81000000e000
>
> which are causing read problems.
>
> crash: read error: kernel virtual address: ffff81000000fa90 type:
> "pglist_data node_next"
>
> I am not sure what these address are and if they are valid.
> Is there a way to verify these addresses, through gdb or /dev/kmem
> or something like that ?
>
> Thanks,
> Badari
>
> Here is bottom line we need to understand to fix
> the problem.
>
> 2.6.10:
> pgdat: 0x1000000e000
>
> 2.6.14-rc5:
> pgdat: 0xffff81000000e000
Exactly.
On a 2.6.9 kernel, if you do an nm -Bn on the vmlinux file, you'll
first
see a bunch of "A" type absolute symbols, followed by the text
symbols, then readonly data, data, and so on. Eventually you'll
bump into the per-cpu symbols:
$ nm -Bn vmlinux
0000000000088861 A __crc_dev_mc_delete
000000000014bfd1 A __crc_smp_call_function
00000000002de2e0 A __crc___skb_linearize
0000000000442f14 A __crc_tty_register_device
000000000060e766 A __crc_tty_termios_baud_rate
0000000000712c54 A __crc_remove_inode_hash
00000000007f8e0b A __crc_xfrm_policy_alloc
0000000000801678 A __crc_flush_scheduled_work
0000000000a64d75 A __crc_neigh_changeaddr
... <snip>
00000000ffdf0b3d A __crc_usb_driver_release_interface
00000000ffe031fc A __crc_udp_proc_unregister
00000000ffead192 A __crc_cdrom_number_of_slots
00000000fff9536b A __crc_sock_no_recvmsg
00000000fffb8df8 A __crc_device_unregister
ffffffff80100000 t startup_32
ffffffff80100000 A _text
ffffffff80100081 t reach_compatibility_mode
ffffffff8010008e t second
ffffffff80100100 t reach_long64
ffffffff8010013d T initial_code
ffffffff80100145 T init_rsp
ffffffff80100150 T no_long_mode
ffffffff80100f00 T pGDT32
ffffffff80100f10 t ljumpvector
ffffffff80100f18 T stext
ffffffff80100f18 T _stext
ffffffff80101000 T init_level4_pgt
ffffffff80102000 T level3_ident_pgt
... <snip>
ffffffff80502100 D per_cpu__init_tss
ffffffff80502200 d per_cpu__prof_old_multiplier
ffffffff80502204 d per_cpu__prof_multiplier
ffffffff80502208 d per_cpu__prof_counter
ffffffff80502220 D per_cpu__mmu_gathers
ffffffff80503280 D per_cpu__kstat
ffffffff80503680 d per_cpu__runqueues
ffffffff805048e0 d per_cpu__cpu_domains
ffffffff80504940 d per_cpu__phys_domains
ffffffff805049a0 d per_cpu__node_domains
ffffffff805049f8 D per_cpu__process_counts
ffffffff80504a00 d per_cpu__tasklet_hi_vec
ffffffff80504a08 d per_cpu__tasklet_vec
ffffffff80504a10 d per_cpu__ksoftirqd
ffffffff80504a80 d per_cpu__tvec_bases
ffffffff80506b00 D per_cpu__rcu_bh_data
ffffffff80506b60 D per_cpu__rcu_data
ffffffff80506bc0 d per_cpu__rcu_tasklet
...
So for any data that was specifically created per-cpu,
the symbol above is the starting point, but to get to
the per-cpu structure, the offset value from the
cpu_data.data_offset needs to be applied.
What I don't understand is where the 0xffff810000000000
addresses come into play. Are you seeing them as actual
symbols?
Dave
It looks like level page table changed the layout. Now,
0xffff810000000000 is a valid.
Documentation/x86_64/mm.txt
Virtual memory map with 4 level page tables:
0000000000000000 - 00007fffffffffff (=47bits) user space, different per
mm
hole caused by [48:63] sign extension
ffff800000000000 - ffff80ffffffffff (=40bits) guard hole
ffff810000000000 - ffffc0ffffffffff (=46bits) direct mapping of phys.
memory
ffffc10000000000 - ffffc1ffffffffff (=40bits) hole
ffffc20000000000 - ffffe1ffffffffff (=45bits) vmalloc/ioremap space
... unused hole ...
ffffffff80000000 - ffffffff82800000 (=40MB) kernel text mapping, from
phys 0
... unused hole ...
ffffffff88000000 - fffffffffff00000 (=1919MB) module mapping space
Thanks,
Badari
1:42 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Badari Pulavarty wrote:
It looks like level page table changed the layout. Now,
0xffff810000000000 is a valid.
Documentation/x86_64/mm.txt
Virtual memory map with 4 level page tables:
0000000000000000 - 00007fffffffffff (=47bits) user space, different per
mm
hole caused by [48:63] sign extension
ffff800000000000 - ffff80ffffffffff (=40bits) guard hole
ffff810000000000 - ffffc0ffffffffff (=46bits) direct mapping of phys.
memory
ffffc10000000000 - ffffc1ffffffffff (=40bits) hole
ffffc20000000000 - ffffe1ffffffffff (=45bits) vmalloc/ioremap space
... unused hole ...
ffffffff80000000 - ffffffff82800000 (=40MB) kernel text mapping, from
phys 0
... unused hole ...
ffffffff88000000 - fffffffffff00000 (=1919MB) module mapping space
Thanks,
Badari
Yep. (our last messages crossed...)
There probably will need to be some work done to deal with
page table walkthroughs in x86_64_kvtop() and x86_64_uvtop()
for translating non-unity-mapped memory like user space
and vmalloc space.
But just getting the basic unity-mapped memory accesses will
be a initial necessity before even looking into that.
Dave
12:44 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Here is bottom line we need to understand to fix
the problem.
2.6.10:
pgdat: 0x1000000e000
2.6.14-rc5:
pgdat: 0xffff81000000e000
Thanks,
Badari
1:09 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Okay, I did a crude fix and "crash" now works fine :)
It looks like, we need to teach crash/gdb about
ffff81010bca1100
kind of address for x86-64.
Here is the hack, I did in memory.c:
readmem(ulonglong addr, int memtype, void *buffer, long size,
char *type, ulong error_handle)
{
....
case KVADDR:
if (LKCD_DUMPFILE())
addr = fix_lkcd_address(addr);
if (!((addr & 0xffffffff00000000) ==
0xffffffff00000000)) {
addr = addr & 0x00000fffffffffff;
}
if (!IS_KVADDR(addr)) {
if (PRINT_ERROR_MESSAGE)
error(INFO, INVALID_KVADDR, addr, type);
goto readmem_error;
}
break;
}
Crash output:
=============
elm3a242:~/crash-4.0-2.8 # ./crash
crash 4.0-2.8
Copyright (C) 2002, 2003, 2004, 2005 Red Hat, Inc.
Copyright (C) 2004, 2005 IBM Corporation
Copyright (C) 1999-2005 Hewlett-Packard Co
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: cannot access vmalloc'd module memory
KERNEL: /usr/src/linux-2.6.14-rc5/vmlinux
DUMPFILE: /dev/mem
CPUS: 2
DATE: Thu Oct 27 06:02:01 2005
UPTIME: 00:04:53
LOAD AVERAGE: 0.18, 0.08, 0.07
TASKS: 60
NODENAME: elm3a242
RELEASE: 2.6.14-rc5
VERSION: #1 SMP PREEMPT Thu Oct 27 05:27:10 PDT 2005
MACHINE: x86_64 (3000 Mhz)
MEMORY: 4.6 GB
PID: 11678
COMMAND: "crash"
TASK: ffff810124263760 [THREAD_INFO: ffff81010b584000]
CPU: 1
STATE: TASK_RUNNING (ACTIVE)
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 ffffffff8050ba80 RU 0.0 0 0 [swapper]
0 1 1 ffff810037ea4080 RU 0.0 0 0 [swapper]
1 0 1 ffff810037e974e0 IN 0.0 716 312 init
2 1 0 ffff810037e96e00 IN 0.0 0 0
[migration/0]
3 1 0 ffff810037e96720 IN 0.0 0 0
[ksoftirqd/0]
4 1 0 ffff810037e96040 IN 0.0 0 0
[watchdog/0]
5 1 1 ffff810037ea5520 IN 0.0 0 0
[migration/1]
6 1 1 ffff810037ea4e40 IN 0.0 0 0
[ksoftirqd/1]
7 1 1 ffff810037ea4760 IN 0.0 0 0
[watchdog/1]
8 1 0 ffff8100d7e1b560 IN 0.0 0 0
[events/0]
9 1 1 ffff8100d7e1ae80 IN 0.0 0 0
[events/1]
10 1 0 ffff8100d7e1a7a0 IN 0.0 0 0 [khelper]
11 1 0 ffff8100d7e1a0c0 IN 0.0 0 0 [kthread]
15 11 0 ffff8100d7e495a0 IN 0.0 0 0
[kblockd/0]
16 11 1 ffff8100d7e48ec0 IN 0.0 0 0
[kblockd/1]
19 11 0 ffff8100d7e487e0 IN 0.0 0 0 [khubd]
122 11 0 ffff8100d7e48100 IN 0.0 0 0 [pdflush]
123 11 1 ffff8100d7dff5e0 IN 0.0 0 0 [pdflush]
124 1 1 ffff810037c05620 IN 0.0 0 0 [kswapd0]
125 11 0 ffff8100d7dfef00 IN 0.0 0 0 [aio/0]
126 11 1 ffff8100d7dfe820 IN 0.0 0 0 [aio/1]
200 11 0 ffff8100d7dfe140 IN 0.0 0 0 [kseriod]
259 11 0 ffff810037c91660 IN 0.0 0 0
[scsi_eh_0]
292 11 0 ffff810037c901c0 IN 0.0 0 0
[scsi_eh_1]
570 1 1 ffff810127751760 IN 0.0 0 0
[kjournald]
1177 1 1 ffff810037c04f40 IN 0.0 2532 464
irqbalance
2274 1 0 ffff810122ff38a0 IN 0.0 3608 760 syslogd
2316 1 0 ffff810122ca49a0 IN 0.0 2664 736 klogd
2361 1 0 ffff81012529a380 IN 0.0 3584 388 resmgrd
2363 1 0 ffff810122875860 IN 0.0 4620 500 portmap
2403 1 0 ffff8101232cd6a0 IN 0.0 12700 1328 slpd
2505 1 1 ffff810123a2c100 IN 0.1 24152 6636 cupsd
2601 1 0 ffff810122ff31c0 IN 0.0 26824 1788 sshd
2615 1 1 ffff810122ca77e0 IN 0.1 71188 4240 slapd
2622 1 0 ffff810122ca6a20 IN 0.1 71188 4240 slapd
2825 1 0 ffff810120c5d8e0 IN 0.1 71188 4240 slapd
3113 1 0 ffff810124d98e00 IN 0.0 2544 600 hwscand
4120 1 0 ffff810123c68a20 IN 0.0 20080 2344 master
4196 4120 0 ffff81011d9c07a0 IN 0.0 20140 2296 pickup
4197 4120 0 ffff81011d9c00c0 IN 0.0 20188 2348 qmgr
9513 1 1 ffff8101099a77a0 IN 0.0 48912 912 nscd
9514 1 1 ffff8101099a69e0 IN 0.0 48912 912 nscd
9515 1 1 ffff8101099a6300 IN 0.0 48912 912 nscd
9516 1 0 ffff8101095777e0 IN 0.0 48912 912 nscd
9517 1 0 ffff810109577100 IN 0.0 48912 912 nscd
9518 1 0 ffff810109576a20 IN 0.0 48912 912 nscd
9525 1 1 ffff81010a49e440 IN 0.0 2544 632 cron
10329 1 0 ffff810109dfa920 IN 0.0 8984 768 kdm
10347 10329 1 ffff81010ab11860 IN 0.6 40656 27528 X
10348 10329 0 ffff81010af477a0 ?? 0.0 12200 1356 kdm
10388 1 0 ffff81012762e9e0 IN 0.0 2760 724 mingetty
10389 1 0 ffff8101242622c0 IN 0.0 2756 708 mingetty
10390 1 0 ffff81010cabce40 IN 0.0 2756 716 mingetty
10392 1 0 ffff81010c2bb0c0 IN 0.0 2760 724 mingetty
10393 1 1 ffff81011fd176e0 IN 0.0 2756 708 mingetty
10394 1 1 ffff81010bca1100 IN 0.0 2760 724 mingetty
10967 10348 1 ffff81011edee760 IN 0.4 69108 18472 kdm_greet
10979 2601 0 ffff8101099a70c0 IN 0.1 40268 3028 sshd
10981 10979 0 ffff81010ab103c0 IN 0.1 8432 2864 bash
11678 10981 1 ffff810124263760 RU 3.0 86304 147740 crash
11690 11678 0 ffff81010c2ba9e0 RU 0.0 0 0 crash
crash>
crash>
crash>
crash>
1:36 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Badari Pulavarty wrote:
Okay, I did a crude fix and "crash" now works fine :)
Well - not so fast! I wouldn't call it "just fine" just yet! ;-)
See below...
It looks like, we need to teach crash/gdb about
ffff81010bca1100
kind of address for x86-64.
Here is the hack, I did in memory.c:
readmem(ulonglong addr, int memtype, void *buffer, long size,
char *type, ulong error_handle)
{
....
case KVADDR:
if (LKCD_DUMPFILE())
addr = fix_lkcd_address(addr);
if (!((addr & 0xffffffff00000000) ==
0xffffffff00000000)) {
addr = addr & 0x00000fffffffffff;
}
if (!IS_KVADDR(addr)) {
if (PRINT_ERROR_MESSAGE)
error(INFO, INVALID_KVADDR, addr, type);
goto readmem_error;
}
break;
}
Crash output:
=============
elm3a242:~/crash-4.0-2.8 # ./crash
crash 4.0-2.8
Copyright (C) 2002, 2003, 2004, 2005 Red Hat, Inc.
Copyright (C) 2004, 2005 IBM Corporation
Copyright (C) 1999-2005 Hewlett-Packard Co
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: cannot access vmalloc'd module memory
That's not so good -- again, see below...
KERNEL: /usr/src/linux-2.6.14-rc5/vmlinux
DUMPFILE: /dev/mem
CPUS: 2
DATE: Thu Oct 27 06:02:01 2005
UPTIME: 00:04:53
LOAD AVERAGE: 0.18, 0.08, 0.07
TASKS: 60
NODENAME: elm3a242
RELEASE: 2.6.14-rc5
VERSION: #1 SMP PREEMPT Thu Oct 27 05:27:10 PDT 2005
MACHINE: x86_64 (3000 Mhz)
MEMORY: 4.6 GB
PID: 11678
COMMAND: "crash"
TASK: ffff810124263760 [THREAD_INFO: ffff81010b584000]
CPU: 1
STATE: TASK_RUNNING (ACTIVE)
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 ffffffff8050ba80 RU 0.0 0 0 [swapper]
0 1 1 ffff810037ea4080 RU 0.0 0 0 [swapper]
1 0 1 ffff810037e974e0 IN 0.0 716 312 init
2 1 0 ffff810037e96e00 IN 0.0 0 0
[migration/0]
3 1 0 ffff810037e96720 IN 0.0 0 0
[ksoftirqd/0]
4 1 0 ffff810037e96040 IN 0.0 0 0
[watchdog/0]
5 1 1 ffff810037ea5520 IN 0.0 0 0
[migration/1]
6 1 1 ffff810037ea4e40 IN 0.0 0 0
[ksoftirqd/1]
7 1 1 ffff810037ea4760 IN 0.0 0 0
[watchdog/1]
8 1 0 ffff8100d7e1b560 IN 0.0 0 0
[events/0]
9 1 1 ffff8100d7e1ae80 IN 0.0 0 0
[events/1]
10 1 0 ffff8100d7e1a7a0 IN 0.0 0 0 [khelper]
11 1 0 ffff8100d7e1a0c0 IN 0.0 0 0 [kthread]
15 11 0 ffff8100d7e495a0 IN 0.0 0 0
[kblockd/0]
16 11 1 ffff8100d7e48ec0 IN 0.0 0 0
[kblockd/1]
19 11 0 ffff8100d7e487e0 IN 0.0 0 0 [khubd]
122 11 0 ffff8100d7e48100 IN 0.0 0 0 [pdflush]
123 11 1 ffff8100d7dff5e0 IN 0.0 0 0 [pdflush]
124 1 1 ffff810037c05620 IN 0.0 0 0 [kswapd0]
125 11 0 ffff8100d7dfef00 IN 0.0 0 0 [aio/0]
126 11 1 ffff8100d7dfe820 IN 0.0 0 0 [aio/1]
200 11 0 ffff8100d7dfe140 IN 0.0 0 0 [kseriod]
259 11 0 ffff810037c91660 IN 0.0 0 0
[scsi_eh_0]
292 11 0 ffff810037c901c0 IN 0.0 0 0
[scsi_eh_1]
570 1 1 ffff810127751760 IN 0.0 0 0
[kjournald]
1177 1 1 ffff810037c04f40 IN 0.0 2532 464
irqbalance
2274 1 0 ffff810122ff38a0 IN 0.0 3608 760 syslogd
2316 1 0 ffff810122ca49a0 IN 0.0 2664 736 klogd
2361 1 0 ffff81012529a380 IN 0.0 3584 388 resmgrd
2363 1 0 ffff810122875860 IN 0.0 4620 500 portmap
2403 1 0 ffff8101232cd6a0 IN 0.0 12700 1328 slpd
2505 1 1 ffff810123a2c100 IN 0.1 24152 6636 cupsd
2601 1 0 ffff810122ff31c0 IN 0.0 26824 1788 sshd
2615 1 1 ffff810122ca77e0 IN 0.1 71188 4240 slapd
2622 1 0 ffff810122ca6a20 IN 0.1 71188 4240 slapd
2825 1 0 ffff810120c5d8e0 IN 0.1 71188 4240 slapd
3113 1 0 ffff810124d98e00 IN 0.0 2544 600 hwscand
4120 1 0 ffff810123c68a20 IN 0.0 20080 2344 master
4196 4120 0 ffff81011d9c07a0 IN 0.0 20140 2296 pickup
4197 4120 0 ffff81011d9c00c0 IN 0.0 20188 2348 qmgr
9513 1 1 ffff8101099a77a0 IN 0.0 48912 912 nscd
9514 1 1 ffff8101099a69e0 IN 0.0 48912 912 nscd
9515 1 1 ffff8101099a6300 IN 0.0 48912 912 nscd
9516 1 0 ffff8101095777e0 IN 0.0 48912 912 nscd
9517 1 0 ffff810109577100 IN 0.0 48912 912 nscd
9518 1 0 ffff810109576a20 IN 0.0 48912 912 nscd
9525 1 1 ffff81010a49e440 IN 0.0 2544 632 cron
10329 1 0 ffff810109dfa920 IN 0.0 8984 768 kdm
10347 10329 1 ffff81010ab11860 IN 0.6 40656 27528 X
10348 10329 0 ffff81010af477a0 ?? 0.0 12200 1356 kdm
10388 1 0 ffff81012762e9e0 IN 0.0 2760 724 mingetty
10389 1 0 ffff8101242622c0 IN 0.0 2756 708 mingetty
10390 1 0 ffff81010cabce40 IN 0.0 2756 716 mingetty
10392 1 0 ffff81010c2bb0c0 IN 0.0 2760 724 mingetty
10393 1 1 ffff81011fd176e0 IN 0.0 2756 708 mingetty
10394 1 1 ffff81010bca1100 IN 0.0 2760 724 mingetty
10967 10348 1 ffff81011edee760 IN 0.4 69108 18472 kdm_greet
10979 2601 0 ffff8101099a70c0 IN 0.1 40268 3028 sshd
10981 10979 0 ffff81010ab103c0 IN 0.1 8432 2864 bash
> 11678 10981 1 ffff810124263760 RU 3.0 86304 147740 crash
> 11690 11678 0 ffff81010c2ba9e0 RU 0.0 0 0 crash
crash>
Ok -- so there has been a wholesale change in the virtual address
space ranges used. In 2.6.9 (and 2.4 as well), you see:
crash> mach
MACHINE TYPE: x86_64
MEMORY SIZE: 1 GB
CPUS: 4
PROCESSOR SPEED: 2793 Mhz
HZ: 1000
PAGE SIZE: 4096
L1 CACHE SIZE: 64
KERNEL VIRTUAL BASE: 10000000000
KERNEL VMALLOC BASE: ffffff0000000000
KERNEL START MAP: ffffffff80000000
KERNEL MODULES BASE: ffffffffa0000000
KERNEL STACK SIZE: 8192
crash>
...and there's no unity mapping for anything in the "new" range
like those seen above for the task addresses (ffff81...). And
the fact that you can't access modules also implies that something
there has changed as well.
The values for all of the above are pretty much hard-coded,
and now need to be dynamically determined. Note crash's "defs.h"
has these values in place, which have been the case for 2.4
and 2.6 kernels -- well, at least until 2.6.10:
#ifdef X86_64
#define _64BIT_
#define MACHINE_TYPE "X86_64"
#define USERSPACE_TOP 0x0000008000000000
#define __START_KERNEL_map 0xffffffff80000000
#define PAGE_OFFSET 0x0000010000000000
#define VMALLOC_START 0xffffff0000000000
#define VMALLOC_END 0xffffff7fffffffff
#define MODULES_VADDR 0xffffffffa0000000
#define MODULES_END 0xffffffffafffffff
#define MODULES_LEN (MODULES_END - MODULES_VADDR)
So I believe the place to start would be to make these
values into x86_64-specific variables that get initialized
early on based upon the symbol values gathered during
symtab_init(), which is called by main(). After it
completes, machdep_init(PRE_GDB) is called, i.e. x86_64_init():
/*
* Initialize various subsystems.
*/
fd_init();
buf_init();
cmdline_init();
mem_init();
machdep_init(PRE_SYMTAB);
symtab_init();
machdep_init(PRE_GDB);
kernel_init(PRE_GDB);
verify_version();
datatype_init();
In x86_64_init(PRE_GDB), the former hardwired #defines would need
to be variables, initialized properly based upon clues in the symbol
list.
Interested in taking a look into this?
Dave
3:56 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
On Thu, 2005-10-27 at 14:36 -0400, Dave Anderson wrote:
#ifdef X86_64
#define _64BIT_
#define MACHINE_TYPE "X86_64"
#define USERSPACE_TOP 0x0000008000000000
#define __START_KERNEL_map 0xffffffff80000000
#define PAGE_OFFSET 0x0000010000000000
#define VMALLOC_START 0xffffff0000000000
#define VMALLOC_END 0xffffff7fffffffff
#define MODULES_VADDR 0xffffffffa0000000
#define MODULES_END 0xffffffffafffffff
#define MODULES_LEN (MODULES_END - MODULES_VADDR)
So I believe the place to start would be to make these
values into x86_64-specific variables that get initialized
early on based upon the symbol values gathered during
symtab_init(), which is called by main(). After it
completes, machdep_init(PRE_GDB) is called, i.e. x86_64_init():
/*
* Initialize various subsystems.
*/
fd_init();
buf_init();
cmdline_init();
mem_init();
machdep_init(PRE_SYMTAB);
symtab_init();
machdep_init(PRE_GDB);
kernel_init(PRE_GDB);
verify_version();
datatype_init();
In x86_64_init(PRE_GDB), the former hardwired #defines would need
to be variables, initialized properly based upon clues in the symbol
list.
Interested in taking a look into this?
Dave
Well, I took a stab at it. Here are the changes I made to "defs.h"
looking at Documentation/x86_64/mm.txt. We need to some how put
this under "#if THIS_KERNEL_VERSION > 2.6.10".
--- defs.h.org 2005-10-28 13:43:11.000000000 -0700
+++ defs.h 2005-10-28 13:53:58.000000000 -0700
@@ -1740,14 +1740,14 @@ struct load_module {
#define _64BIT_
#define MACHINE_TYPE "X86_64"
-#define USERSPACE_TOP 0x0000008000000000
+#define USERSPACE_TOP 0x0000800000000000
#define __START_KERNEL_map 0xffffffff80000000
-#define PAGE_OFFSET 0x0000010000000000
+#define PAGE_OFFSET 0xffff810000000000
-#define VMALLOC_START 0xffffff0000000000
-#define VMALLOC_END 0xffffff7fffffffff
-#define MODULES_VADDR 0xffffffffa0000000
-#define MODULES_END 0xffffffffafffffff
+#define VMALLOC_START 0xffffc20000000000
+#define VMALLOC_END 0xffffe1ffffffffff
+#define MODULES_VADDR 0xffffffff88000000
+#define MODULES_END 0xfffffffffff00000
#define MODULES_LEN (MODULES_END - MODULES_VADDR)
#define PTOV(X) ((unsigned long)(X)+(machdep->kvbase))
Even with these changes, I am not sure if crash is running
fine. Its seem doesn't show any useful stacks + there is a
warning on start (about exception stacks).
[root@localhost crash-4.0-2.8]# ./crash
crash 4.0-2.8
Copyright (C) 2002, 2003, 2004, 2005 Red Hat, Inc.
Copyright (C) 2004, 2005 IBM Corporation
Copyright (C) 1999-2005 Hewlett-Packard Co
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: cpu 0 first exception stack: cccccccccccccccc
boot_exception_stacks: ffffffff8052ce80
KERNEL: /usr/src/linux-2.6.14-rc5-madv/vmlinux
DUMPFILE: /dev/mem
CPUS: 2
DATE: Fri Oct 28 13:58:50 2005
UPTIME: 06:32:12
LOAD AVERAGE: 0.11, 0.10, 0.06
TASKS: 66
NODENAME: localhost.localdomain
RELEASE: 2.6.14-rc5
VERSION: #10 SMP Wed Oct 26 15:58:51 PDT 2005
MACHINE: x86_64 (3000 Mhz)
MEMORY: 4.6 GB
PID: 1460
COMMAND: "crash"
TASK: ffff810122c9f0c0 [THREAD_INFO: ffff810113442000]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
crash>
crash> bt 13939
PID: 13939 TASK: ffff810119123740 CPU: 0 COMMAND: "vi"
#0 [ffff810114535c78] schedule at ffffffff803b12b3
RIP: 000000377c7beb95 RSP: 00007ffffff402d8 RFLAGS: 00010246
RAX: 0000000000000017 RBX: ffffffff8010dc26 RCX: 00007ffffff40388
RDX: 0000000000000000 RSI: 00007ffffff400a0 RDI: 0000000000000001
RBP: 0000000000000000 R8: 0000000000000000 R9: 00007ffffff40020
R10: 00007ffffff40020 R11: 0000000000000246 R12: 000000000058b0e0
R13: 000000000058b0e0 R14: 0000000000000058 R15: 0000000000000001
ORIG_RAX: 0000000000000017 CS: 0033 SS: 002b
It shows only "schedule" for all processes. Doesn't seem to show
any more stack traces.
Thanks,
Badari
4:55 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
Badari Pulavarty wrote:
On Thu, 2005-10-27 at 14:36 -0400, Dave Anderson wrote:
>
>
> #ifdef X86_64
> #define _64BIT_
> #define MACHINE_TYPE "X86_64"
>
> #define USERSPACE_TOP 0x0000008000000000
> #define __START_KERNEL_map 0xffffffff80000000
> #define PAGE_OFFSET 0x0000010000000000
>
> #define VMALLOC_START 0xffffff0000000000
> #define VMALLOC_END 0xffffff7fffffffff
> #define MODULES_VADDR 0xffffffffa0000000
> #define MODULES_END 0xffffffffafffffff
> #define MODULES_LEN (MODULES_END - MODULES_VADDR)
>
> So I believe the place to start would be to make these
> values into x86_64-specific variables that get initialized
> early on based upon the symbol values gathered during
> symtab_init(), which is called by main(). After it
> completes, machdep_init(PRE_GDB) is called, i.e. x86_64_init():
>
> /*
> * Initialize various subsystems.
> */
> fd_init();
> buf_init();
> cmdline_init();
> mem_init();
> machdep_init(PRE_SYMTAB);
> symtab_init();
> machdep_init(PRE_GDB);
> kernel_init(PRE_GDB);
> verify_version();
> datatype_init();
>
> In x86_64_init(PRE_GDB), the former hardwired #defines would need
> to be variables, initialized properly based upon clues in the symbol
> list.
>
> Interested in taking a look into this?
>
> Dave
Well, I took a stab at it. Here are the changes I made to "defs.h"
looking at Documentation/x86_64/mm.txt. We need to some how put
this under "#if THIS_KERNEL_VERSION > 2.6.10".
First off -- thanks very much for all you've done so far. I
really appreciate the effort.
Anyway, what I meant was that -- for x86_64 specifically -- things
like USERSPACE_TOP, PAGE_OFFSET, VMALLOC_START, etc. should no longer
be hardwired #defines, but instead, they should be references to
x86_64 data variables define in x86_64.c. So, for example,
USERSPACE_TOP would be defined something like:
#define USERSPACE_TOP (x86_64_userspace_top)
and there would be one x86_64_xxx variable per virtual address
item. And each of the variables would be initialized in
machdep_init(PRE_GDB), which is called just after symtab_init().
The fact that symtab_init() has been done is important because
the variables behind "THIS_KERNEL_VERSION" haven't even been
initialized yet. So instead, I would look at the symbol_value()
of "_stext", or some known kernel text symbol, and based upon its
value, it would be obvious whether to use the "old" or "new"
virtual address values to then set up the each of the x86_64_xxxx
virtual address values.
But for testing the new addresses, what you've done below should
suffice.
--- defs.h.org 2005-10-28 13:43:11.000000000 -0700
+++ defs.h 2005-10-28 13:53:58.000000000 -0700
@@ -1740,14 +1740,14 @@ struct load_module {
#define _64BIT_
#define MACHINE_TYPE "X86_64"
-#define USERSPACE_TOP 0x0000008000000000
+#define USERSPACE_TOP 0x0000800000000000
#define __START_KERNEL_map 0xffffffff80000000
-#define PAGE_OFFSET 0x0000010000000000
+#define PAGE_OFFSET 0xffff810000000000
-#define VMALLOC_START 0xffffff0000000000
-#define VMALLOC_END 0xffffff7fffffffff
-#define MODULES_VADDR 0xffffffffa0000000
-#define MODULES_END 0xffffffffafffffff
+#define VMALLOC_START 0xffffc20000000000
+#define VMALLOC_END 0xffffe1ffffffffff
+#define MODULES_VADDR 0xffffffff88000000
+#define MODULES_END 0xfffffffffff00000
#define MODULES_LEN (MODULES_END - MODULES_VADDR)
#define PTOV(X) ((unsigned long)(X)+(machdep->kvbase))
Even with these changes, I am not sure if crash is running
fine. Its seem doesn't show any useful stacks + there is a
warning on start (about exception stacks).
I'm wondering whether the per-cpu calculations are being
done correctly? The exception stack addresses come from the
same per-cpu tss_struct code that started this whole mess,
and if the per-cpu address calculations needed to find those data
structures were incorrect, it would lead to exception stack
error message that you're seeing. This is the old code, but if
the readmem() of 7 ebase addresses below came from the
wrong place, the error message you're seeing would result:
} else if (symbol_exists("per_cpu__init_tss")) {
for (c = 0; c < NR_CPUS; c++) {
if ((kt->flags & SMP) && (kt->flags &
PER_CPU_OFF)) {
if (kt->__per_cpu_offset[c] == 0)
break;
vaddr = symbol_value("per_cpu__init_tss") +
kt->__per_cpu_offset[c];
} else
vaddr = symbol_value("per_cpu__init_tss");
vaddr += OFFSET(tss_struct_ist);
readmem(vaddr, KVADDR, &ms->stkinfo.ebase[c][0],
sizeof(ulong) * 7, "tss_struct ist array",
FAULT_ON_ERROR);
if (ms->stkinfo.ebase[c][0] == 0)
break;
}
}
The error message only error checks the contents of cpu 0's array
of exception stack addresses, the first of which should be a
pointer to the "boot_exception_stacks" array in the kernel.
[root@localhost crash-4.0-2.8]# ./crash
crash 4.0-2.8
Copyright (C) 2002, 2003, 2004, 2005 Red Hat, Inc.
Copyright (C) 2004, 2005 IBM Corporation
Copyright (C) 1999-2005 Hewlett-Packard Co
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: cpu 0 first exception stack: cccccccccccccccc
boot_exception_stacks: ffffffff8052ce80
KERNEL: /usr/src/linux-2.6.14-rc5-madv/vmlinux
DUMPFILE: /dev/mem
CPUS: 2
DATE: Fri Oct 28 13:58:50 2005
UPTIME: 06:32:12
LOAD AVERAGE: 0.11, 0.10, 0.06
TASKS: 66
NODENAME: localhost.localdomain
RELEASE: 2.6.14-rc5
VERSION: #10 SMP Wed Oct 26 15:58:51 PDT 2005
MACHINE: x86_64 (3000 Mhz)
MEMORY: 4.6 GB
PID: 1460
COMMAND: "crash"
TASK: ffff810122c9f0c0 [THREAD_INFO: ffff810113442000]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
crash>
crash> bt 13939
PID: 13939 TASK: ffff810119123740 CPU: 0 COMMAND: "vi"
#0 [ffff810114535c78] schedule at ffffffff803b12b3
RIP: 000000377c7beb95 RSP: 00007ffffff402d8 RFLAGS: 00010246
RAX: 0000000000000017 RBX: ffffffff8010dc26 RCX: 00007ffffff40388
RDX: 0000000000000000 RSI: 00007ffffff400a0 RDI: 0000000000000001
RBP: 0000000000000000 R8: 0000000000000000 R9: 00007ffffff40020
R10: 00007ffffff40020 R11: 0000000000000246 R12: 000000000058b0e0
R13: 000000000058b0e0 R14: 0000000000000058 R15: 0000000000000001
ORIG_RAX: 0000000000000017 CS: 0033 SS: 002b
It shows only "schedule" for all processes. Doesn't seem to show
any more stack traces.
I don't really have any suggestions here, other than to determine
why the x86_64_low_budget_back_trace_cmd() section that walks the
process stack is only finding/printing the schedule() line.
Does "bt -t" work?
I note that this one doesn't show the "cannot access vmalloc space"
message. Can you read vmalloc and user space addresses? Does "mod"
work? How about "runq", which is one of the places that depends
upon being able to read per-cpu data?
Thanks (and off for the weekend...),
Dave
Thanks,
Badari
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
5:19 p.m.
New subject: crash 4.0-2.8 fails on 2.6.14-rc5 (EM64T)
On Fri, 2005-10-28 at 17:55 -0400, Dave Anderson wrote:
Badari Pulavarty wrote:
> On Thu, 2005-10-27 at 14:36 -0400, Dave Anderson wrote:
> >
> >
> > #ifdef X86_64
> > #define _64BIT_
> > #define MACHINE_TYPE "X86_64"
> >
> > #define USERSPACE_TOP 0x0000008000000000
> > #define __START_KERNEL_map 0xffffffff80000000
> > #define PAGE_OFFSET 0x0000010000000000
> >
> > #define VMALLOC_START 0xffffff0000000000
> > #define VMALLOC_END 0xffffff7fffffffff
> > #define MODULES_VADDR 0xffffffffa0000000
> > #define MODULES_END 0xffffffffafffffff
> > #define MODULES_LEN (MODULES_END - MODULES_VADDR)
> >
> > So I believe the place to start would be to make these
> > values into x86_64-specific variables that get initialized
> > early on based upon the symbol values gathered during
> > symtab_init(), which is called by main(). After it
> > completes, machdep_init(PRE_GDB) is called, i.e. x86_64_init():
> >
> > /*
> > * Initialize various subsystems.
> > */
> > fd_init();
> > buf_init();
> > cmdline_init();
> > mem_init();
> > machdep_init(PRE_SYMTAB);
> > symtab_init();
> > machdep_init(PRE_GDB);
> > kernel_init(PRE_GDB);
> > verify_version();
> > datatype_init();
> >
> > In x86_64_init(PRE_GDB), the former hardwired #defines would need
> > to be variables, initialized properly based upon clues in the
> symbol
> > list.
> >
> > Interested in taking a look into this?
> >
> > Dave
>
> Well, I took a stab at it. Here are the changes I made to "defs.h"
> looking at Documentation/x86_64/mm.txt. We need to some how put
> this under "#if THIS_KERNEL_VERSION > 2.6.10".
>
>
First off -- thanks very much for all you've done so far. I
really appreciate the effort.
Anyway, what I meant was that -- for x86_64 specifically -- things
like USERSPACE_TOP, PAGE_OFFSET, VMALLOC_START, etc. should no longer
be hardwired #defines, but instead, they should be references to
x86_64 data variables define in x86_64.c. So, for example,
USERSPACE_TOP would be defined something like:
#define USERSPACE_TOP (x86_64_userspace_top)
and there would be one x86_64_xxx variable per virtual address
item. And each of the variables would be initialized in
machdep_init(PRE_GDB), which is called just after symtab_init().
The fact that symtab_init() has been done is important because
the variables behind "THIS_KERNEL_VERSION" haven't even been
initialized yet. So instead, I would look at the symbol_value()
of "_stext", or some known kernel text symbol, and based upon its
value, it would be obvious whether to use the "old" or "new"
virtual address values to then set up the each of the x86_64_xxxx
virtual address values.
But for testing the new addresses, what you've done below should
suffice.
>
There is no simple way to add #if KERNEL_VERSION > 2.6.10
in the header file and leave the hardcoded values there ?
>
> --- defs.h.org 2005-10-28 13:43:11.000000000 -0700
> +++ defs.h 2005-10-28 13:53:58.000000000 -0700
> @@ -1740,14 +1740,14 @@ struct load_module {
> #define _64BIT_
> #define MACHINE_TYPE "X86_64"
>
> -#define USERSPACE_TOP 0x0000008000000000
> +#define USERSPACE_TOP 0x0000800000000000
> #define __START_KERNEL_map 0xffffffff80000000
> -#define PAGE_OFFSET 0x0000010000000000
> +#define PAGE_OFFSET 0xffff810000000000
>
> -#define VMALLOC_START 0xffffff0000000000
> -#define VMALLOC_END 0xffffff7fffffffff
> -#define MODULES_VADDR 0xffffffffa0000000
> -#define MODULES_END 0xffffffffafffffff
> +#define VMALLOC_START 0xffffc20000000000
> +#define VMALLOC_END 0xffffe1ffffffffff
> +#define MODULES_VADDR 0xffffffff88000000
> +#define MODULES_END 0xfffffffffff00000
> #define MODULES_LEN (MODULES_END - MODULES_VADDR)
>
> #define PTOV(X) ((unsigned long)(X)+(machdep-
> >kvbase))
>
> Even with these changes, I am not sure if crash is running
> fine. Its seem doesn't show any useful stacks + there is a
> warning on start (about exception stacks).
>
>
I'm wondering whether the per-cpu calculations are being
done correctly? The exception stack addresses come from the
same per-cpu tss_struct code that started this whole mess,
and if the per-cpu address calculations needed to find those data
structures were incorrect, it would lead to exception stack
error message that you're seeing. This is the old code, but if
the readmem() of 7 ebase addresses below came from the
wrong place, the error message you're seeing would result:
} else if (symbol_exists("per_cpu__init_tss")) {
for (c = 0; c < NR_CPUS; c++) {
if ((kt->flags & SMP) && (kt->flags &
PER_CPU_OFF)) {
if (kt->__per_cpu_offset[c] == 0)
break;
vaddr = symbol_value
("per_cpu__init_tss") +
kt->__per_cpu_offset[c];
} else
vaddr = symbol_value
("per_cpu__init_tss");
vaddr += OFFSET(tss_struct_ist);
readmem(vaddr, KVADDR, &ms->stkinfo.ebase
[c][0],
sizeof(ulong) * 7, "tss_struct ist
array",
FAULT_ON_ERROR);
if (ms->stkinfo.ebase[c][0] == 0)
break;
}
}
The error message only error checks the contents of cpu 0's array
of exception stack addresses, the first of which should be a
pointer to the "boot_exception_stacks" array in the kernel.
I will take a closer look.
>
>
> [root@localhost crash-4.0-2.8]# ./crash
>
> crash 4.0-2.8
> Copyright (C) 2002, 2003, 2004, 2005 Red Hat, Inc.
> Copyright (C) 2004, 2005 IBM Corporation
> Copyright (C) 1999-2005 Hewlett-Packard Co
> Copyright (C) 1999, 2002 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it
> under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb 6.1
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and
> you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for
> details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> WARNING: cpu 0 first exception stack: cccccccccccccccc
> boot_exception_stacks: ffffffff8052ce80
>
> KERNEL: /usr/src/linux-2.6.14-rc5-madv/vmlinux
> DUMPFILE: /dev/mem
> CPUS: 2
> DATE: Fri Oct 28 13:58:50 2005
> UPTIME: 06:32:12
> LOAD AVERAGE: 0.11, 0.10, 0.06
> TASKS: 66
> NODENAME: localhost.localdomain
> RELEASE: 2.6.14-rc5
> VERSION: #10 SMP Wed Oct 26 15:58:51 PDT 2005
> MACHINE: x86_64 (3000 Mhz)
> MEMORY: 4.6 GB
> PID: 1460
> COMMAND: "crash"
> TASK: ffff810122c9f0c0 [THREAD_INFO: ffff810113442000]
> CPU: 0
> STATE: TASK_RUNNING (ACTIVE)
>
> crash>
> crash> bt 13939
> PID: 13939 TASK: ffff810119123740 CPU: 0 COMMAND: "vi"
> #0 [ffff810114535c78] schedule at ffffffff803b12b3
> RIP: 000000377c7beb95 RSP: 00007ffffff402d8 RFLAGS: 00010246
> RAX: 0000000000000017 RBX: ffffffff8010dc26 RCX:
> 00007ffffff40388
> RDX: 0000000000000000 RSI: 00007ffffff400a0 RDI:
> 0000000000000001
> RBP: 0000000000000000 R8: 0000000000000000 R9:
> 00007ffffff40020
> R10: 00007ffffff40020 R11: 0000000000000246 R12:
> 000000000058b0e0
> R13: 000000000058b0e0 R14: 0000000000000058 R15:
> 0000000000000001
> ORIG_RAX: 0000000000000017 CS: 0033 SS: 002b
>
> It shows only "schedule" for all processes. Doesn't seem to show
> any more stack traces.
>
>
I don't really have any suggestions here, other than to determine
why the x86_64_low_budget_back_trace_cmd() section that walks the
process stack is only finding/printing the schedule() line.
Does "bt -t" work?
I note that this one doesn't show the "cannot access vmalloc space"
message. Can you read vmalloc and user space addresses? Does "mod"
work? How about "runq", which is one of the places that depends
upon being able to read per-cpu data?
bt -t seems to better.
crash> bt 3144
PID: 3144 TASK: ffff81011dd1e100 CPU: 0 COMMAND: "mingetty"
#0 [ffff81011d6b9c68] schedule at ffffffff803b12b3
RIP: 000000377c7b85b2 RSP: 00007fffff87a110 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffffffff8010dc26 RCX: 00007fffff87a7b0
RDX: 0000000000000001 RSI: 00007fffff87a8c7 RDI: 0000000000000000
RBP: 00007fffff87aca0 R8: 00002aaaaaac9b00 R9: 0000000000000000
R10: 0000000000000001 R11: 0000000000000246 R12: 00007fffff87a900
R13: 0000000000502d20 R14: 0000000000000000 R15: 000000007c92d8c0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
crash> bt -t 3144
PID: 3144 TASK: ffff81011dd1e100 CPU: 0 COMMAND: "mingetty"
START: thread_return (schedule) at ffffffff803b12b3
[ffff81011d6b9d10] do_con_write at ffffffff802689da
[ffff81011d6b9d80] schedule_timeout at ffffffff803b1e4e
[ffff81011d6b9db0] _spin_lock_irqsave at ffffffff803b28ce
[ffff81011d6b9dc0] add_wait_queue at ffffffff8014cf5c
[ffff81011d6b9de0] read_chan at ffffffff8025d1f7
[ffff81011d6b9e48] default_wake_function at ffffffff80130c90
[ffff81011d6b9e78] default_wake_function at ffffffff80130c90
[ffff81011d6b9e90] tty_ldisc_deref at ffffffff802571c4
[ffff81011d6b9ed0] tty_read at ffffffff802575ee
[ffff81011d6b9f10] vfs_read at ffffffff80183a46
[ffff81011d6b9f40] sys_read at ffffffff80183e03
[ffff81011d6b9f80] system_call at ffffffff8010dc26
RIP: 000000377c7b85b2 RSP: 00007fffff87a110 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffffffff8010dc26 RCX: 00007fffff87a7b0
RDX: 0000000000000001 RSI: 00007fffff87a8c7 RDI: 0000000000000000
RBP: 00007fffff87aca0 R8: 00002aaaaaac9b00 R9: 0000000000000000
R10: 0000000000000001 R11: 0000000000000246 R12: 00007fffff87a900
R13: 0000000000502d20 R14: 0000000000000000 R15: 000000007c92d8c0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
crash>
vmalloc space seems ok:
crash> mod
MODULE NAME SIZE OBJECT FILE
ffffffff88011f80 floppy 77896 (not loaded) [CONFIG_KALLSYMS]
ffffffff8801db80 i2c_core 29056 (not loaded) [CONFIG_KALLSYMS]
ffffffff88022800 i2c_i801 11796 (not loaded) [CONFIG_KALLSYMS]
ffffffff88025900 hw_random 7968 (not loaded) [CONFIG_KALLSYMS]
ffffffff88030500 ehci_hcd 39688 (not loaded) [CONFIG_KALLSYMS]
ffffffff8803ae80 uhci_hcd 38048 (not loaded) [CONFIG_KALLSYMS]
ffffffff88086380 ipv6 309760 (not loaded) [CONFIG_KALLSYMS]
ffffffff8809a380 dm_mod 70232 (not loaded) [CONFIG_KALLSYMS]
ffffffff880a3100 dm_mirror 26504 (not loaded) [CONFIG_KALLSYMS]
ffffffff880cc300 sunrpc 177096 (not loaded) [CONFIG_KALLSYMS]
ffffffff880d8100 autofs4 26376 (not loaded) [CONFIG_KALLSYMS]
ffffffff880e5100 parport 46988 (not loaded) [CONFIG_KALLSYMS]
ffffffff880ea800 lp 17616 (not loaded) [CONFIG_KALLSYMS]
ffffffff880f6c80 parport_pc 33768 (not loaded) [CONFIG_KALLSYMS]
crash> runq
RUNQUEUES[0]: ffff8100050ee6e0
ACTIVE PRIO_ARRAY: ffff8100050ee760
[115] PID: 30383 TASK: ffff810110c720c0 CPU: 0 COMMAND: "crash"
PID: 30227 TASK: ffff810115f1f8c0 CPU: 0 COMMAND: "sshd"
EXPIRED PRIO_ARRAY: ffff8100050ef040
RUNQUEUES[1]: ffff8100050f66e0
ACTIVE PRIO_ARRAY: ffff8100050f6760
[117] PID: 3505 TASK: ffff81011cec4780 CPU: 1 COMMAND: "crash"
EXPIRED PRIO_ARRAY: ffff8100050f7040
Have a nice weekend, we can take a look at it on Monday.
Thanks,
Badari
6889
days inactive
6891
days old
devel@lists.crash-utility.osci.io
22 comments
2 participants
participants (2)
-
Badari Pulavarty -
Dave Anderson