Hi Castor,

Preliminary tests look good on x86_64, and for the most part ia64.

On ia64, some modules never load any .data, .data.rel, etc. sections,
and when that happens the "p" command for their data (d) symbols
doesn't work.  But other modules work just fine?  No regression
though...

On s390x, we don't stand a prayer, because the modules themselves
don't seem to contain any debug info?  It looks like this:

crash> mod -l
     MODULE       NAME                  SIZE  OBJECT FILE
        2081a800  dm_mirror            56184  (not loaded)  [CONFIG_KALLSYMS]
        20826b00  sd_mod               44584  (not loaded)  [CONFIG_KALLSYMS]
        2082f100  dasd_fba_mod         29696  (not loaded)  [CONFIG_KALLSYMS]
        2084c600  jbd                 100688  (not loaded)  [CONFIG_KALLSYMS]
        20867900  dasd_mod            116892  (not loaded)  [CONFIG_KALLSYMS]
        20880600  dasd_eckd_mod        84224  (not loaded)  [CONFIG_KALLSYMS]
        2088d900  dm_zero              19456  (not loaded)  [CONFIG_KALLSYMS]
        208c0b00  ext3                200208  (not loaded)  [CONFIG_KALLSYMS]
        208e9600  scsi_transport_fc    67848  (not loaded)  [CONFIG_KALLSYMS]
        208f4e00  dm_snapshot          41288  (not loaded)  [CONFIG_KALLSYMS]
        2094b900  scsi_mod            233104  (not loaded)  [CONFIG_KALLSYMS]
        20966100  qdio                103768  (not loaded)  [CONFIG_KALLSYMS]
        2097f900  dm_mod               99544  (not loaded)  [CONFIG_KALLSYMS]
        20993c00  ccwgroup             73472  (not loaded)  [CONFIG_KALLSYMS]
        209a5300  cu3088               67080  (not loaded)  [CONFIG_KALLSYMS]
        209f5a00  zfcp                236832  (not loaded)  [CONFIG_KALLSYMS]
        20a4ed00  sg                  151600  (not loaded)  [CONFIG_KALLSYMS]
        20bcbf00  autofs4             254472  (not loaded)  [CONFIG_KALLSYMS]
        20c67600  qeth                563600  (not loaded)  [CONFIG_KALLSYMS]
        20fb0000  sunrpc             1076664  (not loaded)  [CONFIG_KALLSYMS]
        212aa100  ipv6               2300920  (not loaded)  [CONFIG_KALLSYMS]
crash> mod -s ext3
     MODULE       NAME                  SIZE  OBJECT FILE
        208c0b00  ext3                200208  /lib/modules/2.6.18-1.2961.el5/kernel/fs/ext3/ext3.ko
crash> p ext3_file_operations
ext3_file_operations = $1 = 0
crash> whatis ext3_file_operations
<data variable, no debug info> ext3_file_operations;
crash> sym ext3_file_operations
208bff88 (D) ext3_file_operations
crash>

The add-symbol-file line looks OK, although the address of
ext3_file_operations looks like it's in the .data.rel.ro_addr
section (?):

add-symbol-file request add-symbol-file /lib/modules/2.6.18-1.2961.el5/kernel/fs/ext3/ext3.ko 0x208953a8 -s .exit.text 0x208adfc4 -s .rodata 0x208ae000 -s .rodata.str1.2 0x208af1b8 -s __versions 0x208b1d60 -s .data.rel.ro 0x208bfea0 -s .data.rel 0x208c01f8 -s .data.rel.local 0x208c0740 -s .gnu.linkonce.this_module 0x208c0b00 -s .bss 0x208c4e00
add symbol table from file "/lib/modules/2.6.18-1.2961.el5/kernel/fs/ext3/ext3.ko" at
        .text_addr = 0x208953a8
        .exit.text_addr = 0x208adfc4
        .rodata_addr = 0x208ae000
        .rodata.str1.2_addr = 0x208af1b8
        __versions_addr = 0x208b1d60
        .data.rel.ro_addr = 0x208bfea0
        .data.rel_addr = 0x208c01f8
        .data.rel.local_addr = 0x208c0740
        .gnu.linkonce.this_module_addr = 0x208c0b00
        .bss_addr = 0x208c4e00

Anyway, the behavior is identical to what it's always been,
so there's no regression.

I'll keep tinkering as I get time...

Thanks,
  Dave
 

Castor Fu wrote:

 
Finding the overrun wasn't actually that hard.  It's also fixed in the current GDB tree.
I've attached a patch which fixes the problem in symfile.c.  The point of this patch
is to fix loading kernel modules symbol information on 2.6 for those who have not
been following this.
Hopefully this will work on other platforms too....
   -castor
From: crash-utility-bounces@redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf Of Castor Fu
Sent: Wednesday, January 17, 2007 7:49 AM
To: Discussion list for crash utility usage, maintenance and development; Discussion list for crash utility usage, maintenance and development
Subject: RE: [Crash-utility] test results of latest 4.0-3.16.sym.patch (ia64)
 

Hi Dave:

I reproduced the problem on an x86 system by creating a module with a bunch
of sections.

I then found the following in gdb-6.1/gdb/symfile.c:add_symbol_table_command()
 

    num_sec_opts = 16;

with additional code for xreallocing if it turned out to have too many
sections.

This seems to be the code which is broken.  I'm loath to figure out exactly
what it is...  I'll put together a patch against 4.0-3.17 which jacks this
up, and probably print a warning if we exceed the count.

Thanks for digging so far into this.

   -castor

-----Original Message-----
From: crash-utility-bounces@redhat.com on behalf of Dave Anderson
Sent: Thu 1/4/2007 8:22 AM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] test results of latest 4.0-3.16.sym.patch (ia64)
 

Hi Castor,

Another FYI re: the xrealloc() crash.  The problem appears
to be specific to gdb.

I captured the "add-symbol-file" command string and saved
it in an input file.  Then I brought crash up and executed
the input file, which simply passes the suspect command line
directly to gdb, and it crashes on its own:

crash> < /tmp/junk
crash> add-symbol-file /lib/modules/2.6.18-1.2767.el5/kernel/net/ipv6/ipv6.ko 0xa00000021ed605b0 -s .exit.text 0xa00000021edb49a0 -s .rodata 0xa00000021edbd4c8 -s __ksymtab_strings 0xa00000021edbdc08 -s __versions 0xa00000021edbdf98 -s .data 0xa00000021edd6a20 -s .data.rel.ro 0xa00000021edd6c00 -s __ksymtab_gpl 0xa00000021edd6df8 -s __kcrctab_gpl 0xa00000021edd6ed8 -s .data.rel 0xa00000021edd6f48 -s .data.rel.local 0xa00000021ee39940 -s .data.rel.ro.local 0xa00000021ee3a9c0 -s .data.read_mostly 0xa00000021ee3a9e0 -s __ksymtab 0xa00000021ee3aa60 -s __kcrctab 0xa00000021ee3ac30 -s .gnu.linkonce.this_module 0xa00000021ee3ad80 -s .sdata 0xa00000021ee5d730 -s .bss 0xa00000021ee5b000 -s .sbss 0xa00000021ee5e8b8
add_symbol_file_command: calling xrealloc w/argcnt: 49 arg: [0xa00000021ee5d730]...
*** glibc detected *** ./crash: realloc(): invalid next size: 0x6000000001921fe0 ***
======= Backtrace: =========
/lib/libc.so.6.1[0x20000000002f2a70]
/lib/libc.so.6.1(realloc-0x1cb0b0)[0x20000000002f5e20]
./crash(xmrealloc+0x1fffffffffee6e20)[0x40000000003a7d00]
./crash[0x40000000002ff500]
./crash[0x40000000004221e0]
./crash(cmd_func+0x1ffffffffff61610)[0x4000000000422500]
./crash(execute_command+0x1fffffffffee25f0)[0x40000000003a34f0]
./crash(gdb_command_funnel+0x1fffffffffe2feb0)[0x40000000002f0dc0]
./crash(gdb_interface+0x1fffffffffcd7590)[0x40000000001984b0]
./crash(gdb_pass_through+0x1fffffffffcd6cb0)[0x4000000000197be0]
./crash(cmd_gdb+0x2000000000151068)[0x400000000019bbc0]
./crash(exec_command+0x1fffffffffb99db0)[0x400000000005acf0]
./crash(exec_input_file+0x1fffffffffd86d40)[0x4000000000247c90]
./crash[0x400000000005b420]
./crash(exec_command+0x1fffffffffb99e50)[0x400000000005ad90]
./crash(main_loop+0x1fffffffffb9a2e0)[0x400000000005a8e0]
./crash(current_interp_command_loop+0x200000000001fd60)[0x40000000004e0cc0]
./crash[0x40000000003199c0]
./crash[0x400000000039f370]
./crash[0x40000000003a4260]
./crash(catch_errors+0x1fffffffffee33b0)[0x40000000003a4320]
./crash[0x400000000031a930]
./crash[0x400000000039f370]
./crash[0x40000000003a4260]
./crash(catch_errors+0x1fffffffffee33b0)[0x40000000003a4320]
./crash(gdb_main+0x1fffffffffe58960)[0x40000000003198e0]
./crash(gdb_main_entry+0x1fffffffffe589f0)[0x4000000000319980]
./crash(gdb_main_loop+0x1fffffffffcd54d0)[0x4000000000196470]
./crash(main+0x1fffffffffb99820)[0x400000000005a330]
/lib/libc.so.6.1(__libc_start_main-0x2818f0)[0x200000000023f6c0]
./crash(_start+0x1fffffffffb95240)[0x4000000000056200]
======= Memory map: ========
00000000-00004000 r--p 00000000 00:00 0
2000000000000000-2000000000038000 r-xp 00000000 fd:00 10256390           /lib/ld-2.5.so
2000000000044000-2000000000050000 rw-p 00034000 fd:00 10256390           /lib/ld-2.5.so
2000000000050000-2000000000114000 r-xp 00000000 fd:00 10256405           /lib/libm-2.5.so
2000000000114000-2000000000120000 ---p 000c4000 fd:00 10256405           /lib/libm-2.5.so
2000000000120000-2000000000124000 rw-p 000c0000 fd:00 10256405           /lib/libm-2.5.so
2000000000124000-20000000001b0000 r-xp 00000000 fd:00 10883077           /usr/lib/libncurses.so.5.5
20000000001b0000-20000000001bc000 ---p 0008c000 fd:00 10883077           /usr/lib/libncurses.so.5.5
20000000001bc000-20000000001cc000 rw-p 00088000 fd:00 10883077           /usr/lib/libncurses.so.5.5
20000000001cc000-20000000001d0000 rw-p 20000000001cc000 00:00 0
20000000001d0000-20000000001d8000 r-xp 00000000 fd:00 10256403           /lib/libdl-2.5.so
20000000001d8000-20000000001e4000 ---p 00008000 fd:00 10256403           /lib/libdl-2.5.so
20000000001e4000-20000000001e8000 rw-p 00004000 fd:00 10256403           /lib/libdl-2.5.so
20000000001e8000-200000000020c000 r-xp 00000000 fd:00 10882711           /usr/lib/libz.so.1.2.3
200000000020c000-2000000000218000 ---p 00024000 fd:00 10882711           /usr/lib/libz.so.1.2.3
2000000000218000-200000000021c000 rw-p 00020000 fd:00 10882711           /usr/lib/libz.so.1.2.3
200000000021c000-2000000000480000 r-xp 00000000 fd:00 10256397           /lib/libc-2.5.so
2000000000480000-200000000048c000 ---p 00264000 fd:00 10256397           /lib/libc-2.5.so
200000000048c000-2000000000498000 rw-p 00260000 fd:00 10256397           /lib/libc-2.5.so
2000000000498000-20000000004d8000 rw-p 2000000000498000 00:00 0
20000000004d8000-2000000003c1c000 r--p 00000000 fd:00 10882710           /usr/lib/locale/locale-archive
2000000003c1c000-2000000003c2c000 rw-p 2000000003c1c000 00:00 0
2000000003c38000-2000000003c44000 r-xp 00000000 fd:00 10256427           /lib/libthread_db-1.0.so
2000000003c44000-2000000003c50000 ---p 0000c000 fd:00 10256427           /lib/libthread_db-1.0.so
2000000003c50000-2000000003c54000 rw-p 00008000 fd:00 10256427           /lib/libthread_db-1.0.so
2000000003c54000-2000000003c58000 rw-p 2000000003c54000 00:00 0
2000000003c6c000-2000000003da0000 rw-p 2000000003c6c000 00:00 0
2000000003da0000-2000000003dbc000 r-xp 00000000 fd:00 10884674           /usr/lib/libunwind.so.7.0.0
2000000003dbc000-2000000003dc8000 ---p 0001c000 fd:00 10884674           /usr/lib/libunwind.so.7.0.0
2000000003dc8000-2000000003dcc000 rw-p 00018000 fd:00 10884674           /usr/lib/libunwind.so.7.0.0
2000000003dcc000-2000000003df0000 rw-p 2000000003dcc000 00:00 0
2000000003e00000-2000000003e08000 r--s 00000000 fd:00 10977539           /usr/lib/gconv/gconv-modules.cache
2000000003e08000-2000000003e18000 rw-p 2000000003e08000 00:00 0
2000000003e1c000-2000000006edc000 rw-p 2000000003e1c000 00:00 0
2000000006ee8000-2000000006f04000 r-xp 00000000 fd:00 10256386           /lib/libgcc_s-4.1.1-20061130.so.1
2000000006f04000-2000000006f10000 ---p 0001c000 fd:00 10256386           /lib/libgcc_s-4.1.1-20061130.so.1
2000000006f10000-2000000006f14000 rw-p 00018000 fd:00 10256386           /lib/libgcc_s-4.1.1-20061130.so.1
2000000006f14000-2000000006f24000 rw-p 2000000006f14000 00:00 0
2000000008000000-2000000008024000 rw-p 2000000008000000 00:00 0
2000000008024000-200000000c000000 ---p 2000000008024000 00:00 0
4000000000000000-40000000007e0000 r-xp 00000000 fd:00 9633909            /var/tmp/crash-4.0-3.16/crash
600000000000c000-600000000006c000 rw-p 007dc000 fd:00 9633909            /var/tmp/crash-4.0-3.16/crash
600000000006c000-6000000001fc0000 rw-p 600000000006c000 00:00 0          [heap]
60000fff7fffc000-60000fff80004000 rw-p 60000fff7fffc000 00:00 0
60000ffffe068000-60000ffffe0bc000 rw-p 60000ffffe068000 00:00 0          [stack]
a000000000000000-a000000000020000 ---p 00000000 00:00 0                  [vdso]
Aborted
 
 
  

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility