2:15 p.m.
Dave Anderson wrote:
Hi Castor,
Testing this latest patch on an ia64, there is improvement over
the original, but for one particular module, I get an abort that
was generated from glibc that I have never encountered before.
I saw it the first time when running "mod -S", and subsequently
narrowed it down to the ipv6 module. Check this out:
Typically a realloc() error would occur due to an overflow of a previously
allocated buffer which trashed the malloc-bookkeeping. I thought at first
that the supplied command string buffer of 1500 (BUFSIZE) bytes had
overflowed, but that is a stack buffer and is not malloc'd. And we
don't overflow it anyway.
In any case, with a few debugging lines added, here is the command
buffer for the ipv6 add-symbol-file command, which contains 707 bytes:
add-symbol-file /lib/modules/2.6.18-1.2767.el5/kernel/net/ipv6/ipv6.ko 0xa00000021ed605b0
-s .exit.text
0xa00000021edb49a0 -s .rodata 0xa00000021edbd4c8 -s __ksymtab_strings 0xa00000021edbdc08
-s __versions
0xa00000021edbdf98 -s .data 0xa00000021edd6a20 -s .data.rel.ro 0xa00000021edd6c00 -s
__ksymtab_gpl 0xa00000021edd6df8 -s
__kcrctab_gpl 0xa00000021edd6ed8 -s .data.rel 0xa00000021edd6f48 -s .data.rel.local
0xa00000021ee39940 -s
.data.rel.ro.local 0xa00000021ee3a9c0 -s .data.read_mostly 0xa00000021ee3a9e0 -s __ksymtab
0xa00000021ee3aa60 -s
__kcrctab 0xa00000021ee3ac30 -s .gnu.linkonce.this_module 0xa00000021ee3ad80 -s .sdata
0xa00000021ee5d730 -s .bss
0xa00000021ee5b000 -s .sbss 0xa00000021ee5e8b8
and the realloc() crash that caused the abort was made here in
this part of the upper while loop in add_symbol_file_command()
in gdb-6.1/gdb/symfile.c:
if (expecting_sec_addr)
{
sect_opts[section_index].value = arg;
expecting_sec_addr = 0;
if (++section_index > num_sect_opts)
{
num_sect_opts *= 2;
sect_opts = ((struct sect_opt *)
xrealloc (sect_opts,
num_sect_opts
* sizeof (struct sect_opt)));
}
}
else
error ("USAGE: add-symbol-file <filename>
<textaddress> [-mapped] [-readnow] [-s <secname>
<addr>]*");
At the point of failure, the argcnt had gone from 0 to 47, and the
current arg was pointing to "0xa00000021ee5d730", which is the .sdata
section address. This was the first xrealloc() call made from the
loop above -- so it almost made it through the whole command string
without even having to do an xrealloc().
Anyway, that's as far as I took it -- but perhaps you can
artificially re-create an add-symbol-file command with a command
line size approaching the 700-odd bytes that this one consumes and
which has ~36 sections?
I did put a debug printf prior to each call to execute_command() that
shows the strlen of the command buffer and the lm->mod_sections, and
then did a "mod -S" to load all modules. Note that both the 707 bytes
and the number of sections (36) of the ipv6 module were both the largest:
crash> mod -S | grep strlen
strlen(req->buf): 433 mod_sections: 23
strlen(req->buf): 374 mod_sections: 22
strlen(req->buf): 433 mod_sections: 23
strlen(req->buf): 240 mod_sections: 19
strlen(req->buf): 277 mod_sections: 29
strlen(req->buf): 375 mod_sections: 28
strlen(req->buf): 422 mod_sections: 25
strlen(req->buf): 421 mod_sections: 28
strlen(req->buf): 317 mod_sections: 36
strlen(req->buf): 326 mod_sections: 28
strlen(req->buf): 600 mod_sections: 29
strlen(req->buf): 180 mod_sections: 19
strlen(req->buf): 404 mod_sections: 23
strlen(req->buf): 615 mod_sections: 29
strlen(req->buf): 313 mod_sections: 35
strlen(req->buf): 501 mod_sections: 29
strlen(req->buf): 286 mod_sections: 30
strlen(req->buf): 397 mod_sections: 28
strlen(req->buf): 299 mod_sections: 24
strlen(req->buf): 419 mod_sections: 24
strlen(req->buf): 374 mod_sections: 27
strlen(req->buf): 420 mod_sections: 24
strlen(req->buf): 463 mod_sections: 25
strlen(req->buf): 420 mod_sections: 24
strlen(req->buf): 413 mod_sections: 31
strlen(req->buf): 310 mod_sections: 21
strlen(req->buf): 322 mod_sections: 30
strlen(req->buf): 302 mod_sections: 26
strlen(req->buf): 306 mod_sections: 31
strlen(req->buf): 313 mod_sections: 32
strlen(req->buf): 707 mod_sections: 36
*** glibc detected *** ./crash: realloc(): invalid next size: 0x600000000432f430 ***
So the 707-byte ipv6 command string, or maybe its 36 sections,
or a combination of the two, sends crash/gdb into the weeds...
Dave
# ./crash
crash 4.0-3.16
Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "ia64-unknown-linux-gnu"...
KERNEL: /usr/lib/debug/lib/modules/2.6.18-1.2767.el5/vmlinux
DUMPFILE: /dev/mem
CPUS: 64
DATE: Wed Jan 3 10:43:04 2007
UPTIME: 01:40:46
LOAD AVERAGE: 0.15, 0.11, 0.17
TASKS: 629
NODENAME: altix3.lab.boston.redhat.com
RELEASE: 2.6.18-1.2767.el5
VERSION: #1 SMP Wed Nov 29 17:38:14 EST 2006
MACHINE: ia64 (1500 Mhz)
MEMORY: 122.5 GB
PID: 10699
COMMAND: "crash"
TASK: e00000b04e300000 [THREAD_INFO: e00000b04e301040]
CPU: 7
STATE: TASK_RUNNING (ACTIVE)
crash> mod -l
MODULE NAME SIZE OBJECT FILE
a00000021e189c00 ehci_hcd 204860 (not loaded) [CONFIG_KALLSYMS]
a00000021e1bd100 uhci_hcd 185680 (not loaded) [CONFIG_KALLSYMS]
a00000021e1efb00 ohci_hcd 179956 (not loaded) [CONFIG_KALLSYMS]
a00000021e23cb00 dm_zero 134640 (not loaded) [CONFIG_KALLSYMS]
a00000021e287700 jbd 262432 (not loaded) [CONFIG_KALLSYMS]
a00000021e2b9580 sd_mod 170772 (not loaded) [CONFIG_KALLSYMS]
a00000021e303380 qla1280 276848 (not loaded) [CONFIG_KALLSYMS]
a00000021e374300 ext3 414624 (not loaded) [CONFIG_KALLSYMS]
a00000021e3d9600 scsi_mod 387008 (not loaded) [CONFIG_KALLSYMS]
a00000021e418980 mptbase 235792 (not loaded) [CONFIG_KALLSYMS]
a00000021e44c780 scsi_transport_spi 183672 (not loaded) [CONFIG_KALLSYMS]
a00000021e47ec80 mptscsih 176288 (not loaded) [CONFIG_KALLSYMS]
a00000021e4ab700 mptspi 162536 (not loaded) [CONFIG_KALLSYMS]
a00000021e4e1580 scsi_transport_fc 203748 (not loaded) [CONFIG_KALLSYMS]
a00000021e5fcd00 dm_mod 253328 (not loaded) [CONFIG_KALLSYMS]
a00000021e718e80 qla2xxx 1090472 (not loaded) [CONFIG_KALLSYMS]
a00000021e825680 dm_mirror 187608 (not loaded) [CONFIG_KALLSYMS]
a00000021e857480 autofs4 178336 (not loaded) [CONFIG_KALLSYMS]
a00000021e884800 dm_snapshot 167224 (not loaded) [CONFIG_KALLSYMS]
a00000021e8b1a00 lp 156512 (not loaded) [CONFIG_KALLSYMS]
a00000021e8fdc00 cdrom 206776 (not loaded) [CONFIG_KALLSYMS]
a00000021e935400 sg 203464 (not loaded) [CONFIG_KALLSYMS]
a00000021e96f080 ide_cd 211824 (not loaded) [CONFIG_KALLSYMS]
a00000021ea6bf80 tg3 362244 (not loaded) [CONFIG_KALLSYMS]
a00000021eaa6780 parport 208284 (not loaded) [CONFIG_KALLSYMS]
a00000021eacef00 button 144200 (not loaded) [CONFIG_KALLSYMS]
a00000021eb10980 parport_pc 184504 (not loaded) [CONFIG_KALLSYMS]
a00000021eb3e400 vfat 157504 (not loaded) [CONFIG_KALLSYMS]
a00000021eb9a280 fat 239936 (not loaded) [CONFIG_KALLSYMS]
a00000021ecd6880 sunrpc 468360 (not loaded) [CONFIG_KALLSYMS]
a00000021ee3ad80 ipv6 1141140 (not loaded) [CONFIG_KALLSYMS]
a00000021ee9f580 bluetooth 375704 (not loaded) [CONFIG_KALLSYMS]
a00000021eeef880 l2cap 310456 (not loaded) [CONFIG_KALLSYMS]
a00000021ef48680 rfcomm 347144 (not loaded) [CONFIG_KALLSYMS]
a00000021ef97980 hidp 294256 (not loaded) [CONFIG_KALLSYMS]
crash> mod -s ipv6
*** glibc detected *** ./crash: realloc(): invalid next size: 0x6000000001921fc0 ***
======= Backtrace: =========
/lib/libc.so.6.1[0x20000000002f2a70]
/lib/libc.so.6.1(realloc-0x1cb0b0)[0x20000000002f5e20]
./crash(xmrealloc+0x1fffffffffee6c40)[0x40000000003a7b20]
./crash[0x40000000002ff3a0]
./crash[0x4000000000422000]
./crash(cmd_func+0x1ffffffffff61430)[0x4000000000422320]
./crash(execute_command+0x1fffffffffee2410)[0x40000000003a3310]
./crash(gdb_command_funnel+0x1fffffffffe2f900)[0x40000000002f0810]
./crash(gdb_interface+0x1fffffffffcd7590)[0x40000000001984b0]
./crash[0x4000000000235af0]
./crash(load_module_symbols+0x1fffffffffd748f0)[0x4000000000235820]
./crash[0x4000000000175820]
./crash(cmd_mod+0x2000000000129d68)[0x4000000000174930]
./crash(exec_command+0x1fffffffffb99db0)[0x400000000005acf0]
./crash(main_loop+0x1fffffffffb9a2e0)[0x400000000005a8e0]
./crash(current_interp_command_loop+0x200000000001fb90)[0x40000000004e0ae0]
./crash[0x4000000000319820]
./crash[0x400000000039f1d0]
./crash[0x40000000003a4080]
./crash(catch_errors+0x1fffffffffee31e0)[0x40000000003a4140]
./crash[0x400000000031a790]
./crash[0x400000000039f1d0]
./crash[0x40000000003a4080]
./crash(catch_errors+0x1fffffffffee31e0)[0x40000000003a4140]
./crash(gdb_main+0x1fffffffffe587d0)[0x4000000000319740]
./crash(gdb_main_entry+0x1fffffffffe58860)[0x40000000003197e0]
./crash(gdb_main_loop+0x1fffffffffcd54e0)[0x4000000000196470]
./crash(main+0x1fffffffffb99820)[0x400000000005a330]
/lib/libc.so.6.1(__libc_start_main-0x2818e0)[0x200000000023f6c0]
./crash(_start+0x1fffffffffb95250)[0x4000000000056200]
======= Memory map: ========
00000000-00004000 r--p 00000000 00:00 0
2000000000000000-2000000000038000 r-xp 00000000 fd:00 10256390 /lib/ld-2.5.so
2000000000044000-2000000000050000 rw-p 00034000 fd:00 10256390 /lib/ld-2.5.so
2000000000050000-2000000000114000 r-xp 00000000 fd:00 10256405
/lib/libm-2.5.so
2000000000114000-2000000000120000 ---p 000c4000 fd:00 10256405
/lib/libm-2.5.so
2000000000120000-2000000000124000 rw-p 000c0000 fd:00 10256405
/lib/libm-2.5.so
2000000000124000-20000000001b0000 r-xp 00000000 fd:00 10883077
/usr/lib/libncurses.so.5.5
20000000001b0000-20000000001bc000 ---p 0008c000 fd:00 10883077
/usr/lib/libncurses.so.5.5
20000000001bc000-20000000001cc000 rw-p 00088000 fd:00 10883077
/usr/lib/libncurses.so.5.5
20000000001cc000-20000000001d0000 rw-p 20000000001cc000 00:00 0
20000000001d0000-20000000001d8000 r-xp 00000000 fd:00 10256403
/lib/libdl-2.5.so
20000000001d8000-20000000001e4000 ---p 00008000 fd:00 10256403
/lib/libdl-2.5.so
20000000001e4000-20000000001e8000 rw-p 00004000 fd:00 10256403
/lib/libdl-2.5.so
20000000001e8000-200000000020c000 r-xp 00000000 fd:00 10882711
/usr/lib/libz.so.1.2.3
200000000020c000-2000000000218000 ---p 00024000 fd:00 10882711
/usr/lib/libz.so.1.2.3
2000000000218000-200000000021c000 rw-p 00020000 fd:00 10882711
/usr/lib/libz.so.1.2.3
200000000021c000-2000000000480000 r-xp 00000000 fd:00 10256397
/lib/libc-2.5.so
2000000000480000-200000000048c000 ---p 00264000 fd:00 10256397
/lib/libc-2.5.so
200000000048c000-2000000000498000 rw-p 00260000 fd:00 10256397
/lib/libc-2.5.so
2000000000498000-20000000004d8000 rw-p 2000000000498000 00:00 0
20000000004d8000-2000000003c1c000 r--p 00000000 fd:00 10882710
/usr/lib/locale/locale-archive
2000000003c1c000-2000000003c2c000 rw-p 2000000003c1c000 00:00 0
2000000003c38000-2000000003c44000 r-xp 00000000 fd:00 10256427
/lib/libthread_db-1.0.so
2000000003c44000-2000000003c50000 ---p 0000c000 fd:00 10256427
/lib/libthread_db-1.0.so
2000000003c50000-2000000003c54000 rw-p 00008000 fd:00 10256427
/lib/libthread_db-1.0.so
2000000003c54000-2000000003c58000 rw-p 2000000003c54000 00:00 0
2000000003c6c000-2000000003da0000 rw-p 2000000003c6c000 00:00 0
2000000003da0000-2000000003dbc000 r-xp 00000000 fd:00 10884674
/usr/lib/libunwind.so.7.0.0
2000000003dbc000-2000000003dc8000 ---p 0001c000 fd:00 10884674
/usr/lib/libunwind.so.7.0.0
2000000003dc8000-2000000003dcc000 rw-p 00018000 fd:00 10884674
/usr/lib/libunwind.so.7.0.0
2000000003dcc000-2000000003df0000 rw-p 2000000003dcc000 00:00 0
2000000003e00000-2000000003e08000 r--s 00000000 fd:00 10977539
/usr/lib/gconv/gconv-modules.cache
2000000003e08000-2000000003e18000 rw-p 2000000003e08000 00:00 0
2000000003e1c000-2000000006ecc000 rw-p 2000000003e1c000 00:00 0
2000000006ed8000-2000000006ef4000 r-xp 00000000 fd:00 10256386
/lib/libgcc_s-4.1.1-20061130.so.1
2000000006ef4000-2000000006f00000 ---p 0001c000 fd:00 10256386
/lib/libgcc_s-4.1.1-20061130.so.1
2000000006f00000-2000000006f04000 rw-p 00018000 fd:00 10256386
/lib/libgcc_s-4.1.1-20061130.so.1
2000000006f04000-2000000006f14000 rw-p 2000000006f04000 00:00 0
2000000008000000-2000000008024000 rw-p 2000000008000000 00:00 0
2000000008024000-200000000c000000 ---p 2000000008024000 00:00 0
4000000000000000-40000000007e0000 r-xp 00000000 fd:00 9639915
/var/tmp/crash-4.0-3.16/crash
600000000000c000-600000000006c000 rw-p 007dc000 fd:00 9639915
/var/tmp/crash-4.0-3.16/crash
600000000006c000-6000000001ffc000 rw-p 600000000006c000 00:00 0 [heap]
60000fff7fffc000-60000fff80004000 rw-p 60000fff7fffc000 00:00 0
60000ffffecc0000-60000ffffed14000 rw-p 60000ffffecc0000 00:00 0 [stack]
a000000000000000-a000000000020000 ---p 00000000 00:00 0 [vdso]
Aborted
#
So I set debug to 3, and redirected the debug output to a file.
It's big enough (866K) that I don't want to clutter up everybody's
mailbox, so I copied it here:
http://people.redhat.com/anderson/junk
It reproduces the same debug output each time, which gets followed
immediately by the glibc abort.
Maybe it will contain some clues?
Thanks,
Dave
-----------------------------------------------------------------------------------------------------------
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility