Badari Pulavarty wrote:
On Thu, 2005-10-27 at 13:17 -0400, Dave Anderson wrote:
> Badari Pulavarty wrote:
> > > That debug output certainly seems to pinpoint the issue at hand,
> > doesn't it?
> > > Very interesting...
> > >
> > > What's strange is that the usage of the cpu_pda[i].data_offset by
> > the
> > > per_cpu() macro in "include/asm-x86_64/percpu.h" is unchanged.
> > >
> > > It's probably something very simple going on here, but I don't
> > have
> > > any more ideas at this point.
> >
> > This is the reply I got from Andi Kleen..
> >
> > -------- Forwarded Message --------
> > From: Andi Kleen <ak(a)suse.de>
> > To: Badari Pulavarty <pbadari(a)us.ibm.com>
> > Subject: Re: cpu_pda->data_offset changed recently ?
> > Date: Thu, 27 Oct 2005 16:58:54 +0200
> > On Thursday 27 October 2005 16:53, Badari Pulavarty wrote:
> > > Hi Andi,
> > >
> > > I am trying to fix "crash" utility to make it work on
2.6.14-rc5.
> > > (Its running fine on 2.6.10). It looks like crash utility reads
> > > and uses cpu_pda->data_offset values. It looks like there is a
> > > change between 2.6.10 & 2.6.14-rc5 which is causing
"data_offset"
> > > to be huge values - which is causing "crash" to break.
> > >
> > > I added printk() to find out why ? As you can see from following
> > > what changed - Is this expected ? Please let me know.
> >
> > bootmem used to allocate from the end of the direct mapping on NUMA
> > systems. Now it starts at the beginning, often before the
> > kernel .text.
> > This means it is negative. Perfectly legitimate. crash just has to
> > handle it.
> >
> > -Andi
> >
> > --
> >
> That's what I thought it looked like, although the
> x8664_pda.data_offset
> field is an "unsigned long". Anyway, if you take any of the
> per_cpu__xxx
> symbols from the 2.6.14 kernel, subtract a cpu data_offset, does it
> come up
> with a legitimate virtual address?
Unfortunately, I don't know x86-64 kernel virtual address space
well enough to answer your question.
My understanding is x86-64 kernel addresses look something like:
addr: ffffffff80101000
But now (2.6.14-rc5) I do see address like:
pgdat: 0xffff81000000e000
which are causing read problems.
crash: read error: kernel virtual address: ffff81000000fa90 type:
"pglist_data node_next"
I am not sure what these address are and if they are valid.
Is there a way to verify these addresses, through gdb or /dev/kmem
or something like that ?
Thanks,
Badari
Here is bottom line we need to understand to fix
the problem.
2.6.10:
pgdat: 0x1000000e000
2.6.14-rc5:
pgdat: 0xffff81000000e000
Exactly.
On a 2.6.9 kernel, if you do an nm -Bn on the vmlinux file, you'll first
see a bunch of "A" type absolute symbols, followed by the text
symbols, then readonly data, data, and so on. Eventually you'll
bump into the per-cpu symbols:
$ nm -Bn vmlinux
0000000000088861 A __crc_dev_mc_delete
000000000014bfd1 A __crc_smp_call_function
00000000002de2e0 A __crc___skb_linearize
0000000000442f14 A __crc_tty_register_device
000000000060e766 A __crc_tty_termios_baud_rate
0000000000712c54 A __crc_remove_inode_hash
00000000007f8e0b A __crc_xfrm_policy_alloc
0000000000801678 A __crc_flush_scheduled_work
0000000000a64d75 A __crc_neigh_changeaddr
... <snip>
00000000ffdf0b3d A __crc_usb_driver_release_interface
00000000ffe031fc A __crc_udp_proc_unregister
00000000ffead192 A __crc_cdrom_number_of_slots
00000000fff9536b A __crc_sock_no_recvmsg
00000000fffb8df8 A __crc_device_unregister
ffffffff80100000 t startup_32
ffffffff80100000 A _text
ffffffff80100081 t reach_compatibility_mode
ffffffff8010008e t second
ffffffff80100100 t reach_long64
ffffffff8010013d T initial_code
ffffffff80100145 T init_rsp
ffffffff80100150 T no_long_mode
ffffffff80100f00 T pGDT32
ffffffff80100f10 t ljumpvector
ffffffff80100f18 T stext
ffffffff80100f18 T _stext
ffffffff80101000 T init_level4_pgt
ffffffff80102000 T level3_ident_pgt
... <snip>
ffffffff80502100 D per_cpu__init_tss
ffffffff80502200 d per_cpu__prof_old_multiplier
ffffffff80502204 d per_cpu__prof_multiplier
ffffffff80502208 d per_cpu__prof_counter
ffffffff80502220 D per_cpu__mmu_gathers
ffffffff80503280 D per_cpu__kstat
ffffffff80503680 d per_cpu__runqueues
ffffffff805048e0 d per_cpu__cpu_domains
ffffffff80504940 d per_cpu__phys_domains
ffffffff805049a0 d per_cpu__node_domains
ffffffff805049f8 D per_cpu__process_counts
ffffffff80504a00 d per_cpu__tasklet_hi_vec
ffffffff80504a08 d per_cpu__tasklet_vec
ffffffff80504a10 d per_cpu__ksoftirqd
ffffffff80504a80 d per_cpu__tvec_bases
ffffffff80506b00 D per_cpu__rcu_bh_data
ffffffff80506b60 D per_cpu__rcu_data
ffffffff80506bc0 d per_cpu__rcu_tasklet
...
So for any data that was specifically created per-cpu,
the symbol above is the starting point, but to get to
the per-cpu structure, the offset value from the
cpu_data.data_offset needs to be applied.
What I don't understand is where the 0xffff810000000000
addresses come into play. Are you seeing them as actual
symbols?
Dave