2:44 a.m.
Hi Kazu,
Thanks a lot for your effort and comments. Sorry I was working on some
other tasks yesterday and today. I will look into it later when my
task is done. Thanks again for your help!
Thanks,
Tao Liu
On Thu, Nov 9, 2023 at 4:37 PM HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab(a)nec.com> wrote:
On 2023/11/08 17:16, HAGIO KAZUHITO(萩尾 一仁) wrote:
> On 2023/11/08 12:01, HAGIO KAZUHITO(萩尾 一仁) wrote:
>> Hi Tao,
>>
>> thank you for the information.
>>
>> I'm looking into it, I noticed that the unexpected symbol
"floopy_module_init"
>> is in section .init.text. Crash side doesn't have the symbol info,
probably
>> the address is already freed and reused by the virtio_blk module?
>>
>> crash> mod -S
>> ...
>> crash> sym -m floppy | grep MODULE
>> ffffffffc0092000 MODULE START: floppy
>> ffffffffc00a2f29 MODULE END: floppy
>> crash> sym -m virtio_blk | grep MODULE
>> ffffffffc00a4000 MODULE START: virtio_blk
>> ffffffffc00a86ec MODULE END: virtio_blk
>> crash> gdb info symbol floppy_module_init
>> init_module in section .init.text of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug
>> virtblk_freeze + 33 in section .text of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/virtio_blk.ko.debug
>> crash> gdb info address floppy_module_init
>> Symbol "floppy_module_init" is a function at address
0xffffffffc00a4131.
>> crash> sym floppy_module_init
>> symbol not found: floppy_module_init
>> possible alternatives:
>> (none found)
>>
>> So still not sure, but this might be an issue of handling the init
>> section/symbols of modules..
>
> Hmm, it looks like several sections are mapped outside of the current
> module range. What if they are not specified or they are mapped to
> e.g. an unused range..? or any idea?
First, "mod -S -r" can reproduce this without the patch 1/2 or the commands.
(If the root cause is fixed, I think no need to apply the patch 1/2.)
crash-ups> mod -S -r
crash-ups> p virtio_mq_ops
virtio_mq_ops = $1 = {
queue_rq = 0xffffffffc00a45b0 <floppy_module_init+1151>,
map_queue = 0xffffffff813015c0 <blk_mq_map_queue>,
The following trial patch, which does not map the .init.text section
specifically, fixes the issue.
--- a/symbols.c
+++ b/symbols.c
@@ -13283,7 +13283,7 @@ add_symbol_file_kallsyms(struct load_module *lm, struct
gnu_request *req)
shift_string_right(req->buf, strlen(buf));
BCOPY(buf, req->buf, strlen(buf));
retval = TRUE;
- } else {
+ } else if (!STREQ(section_name, ".init.text")) {
sprintf(buf, " -s %s 0x%lx", section_name,
section_vaddr);
while ((len + strlen(buf)) >= buflen) {
RESIZEBUF(req->buf, buflen, buflen * 2);
crash-dev> mod -S -r
crash-dev> p virtio_mq_ops
virtio_mq_ops = $1 = {
queue_rq = 0xffffffffc00a45b0 <virtio_queue_rq>,
map_queue = 0xffffffff813015c0 <blk_mq_map_queue>,
{
In this way, it looks like the .init.text sections of modules are
mapped to 0x0, but many other sections like .modinfo are mapped here
even without the trial patch.
crash-dev> gdb info address floppy_module_init
Symbol "floppy_module_init" is a function at address 0x131.
crash-dev> gdb info symbol floppy_module_init
irq_stack_union + 305 in section .data..percpu of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/vmlinux
local_init + 305 in section .init.text of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/md/dm-mod.ko.debug
____versions + 49 in section __versions of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/md/dm-region-hash.ko.debug
____versions + 145 in section __versions of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/virtio/virtio.ko.debug
__UNIQUE_ID_vermagic5 + 30 in section .modinfo of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/md/dm-log.ko.debug
__UNIQUE_ID_srcversion7 + 1 in section .modinfo of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/md/dm-mirror.ko.debug
__UNIQUE_ID_alias7 + 30 in section .modinfo of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/virtio/virtio_pci.ko.debug
____versions + 145 in section __versions of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/virtio/virtio_ring.ko.debug
__UNIQUE_ID_alias16 + 36 in section .modinfo of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/ata/ata_generic.ko.debug
____versions + 17 in section __versions of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/fs/mbcache.ko.debug
__UNIQUE_ID_vermagic5 + 45 in section .modinfo of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/net/virtio_net.ko.debug
__UNIQUE_ID_vermagic5 + 44 in section .modinfo of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/cdrom/cdrom.ko.debug
__UNIQUE_ID_vermagic5 + 14 in section .modinfo of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/scsi/sr_mod.ko.debug
__module_depends + 1 in section .modinfo of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/net/ipv4/netfilter/ip_tables.ko.debug
____versions + 81 in section __versions of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/fs/binfmt_misc.ko.debug
init_module in section .init.text of
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug
...
If a module already does not have its init memory range, it might be
a bit better to not specify "-s .init.text <addr>" to add-symbol-file..
Thanks,
Kazu
>
> crash-dev> sym -m floppy | grep MODULE
> ffffffffc0092000 MODULE START: floppy
> ffffffffc00a2f29 MODULE END: floppy
> crash-dev> set debug 1
> debug: 1
> crash-dev> mod -s floppy
> ffffffffc009a000 .note.gnu.build-id
> ffffffffc0092000 .text
> ffffffffc00a4000 .init.text <<< higher than MODULE END
> ffffffffc009971d .text.unlikely
> ffffffffc00998ea .exit.text
> ffffffffc009a040 .rodata
> ffffffffc009a4d0 .rodata.str1.1
> ffffffffc009ad98 .rodata.str1.8
> ffffffffc009be14 .smp_locks
> ffffffffc009be98 .parainstructions
> ffffffffc009beb8 __param
> ffffffffc009bf18 __mcount_loc
> ffffffffc009d000 .data
> ffffffffc00a5000 .init.data <<<
> ffffffffc009db00 .gnu.linkonce.this_module
> ffffffffc009dd40 .bss
> ffffffffc00a6000 .symtab <<<
> ffffffffc00a80b8 .strtab <<<
> add-symbol-file
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug
0xffffffffc0092000 -s .note.gnu.build-id 0xffffffffc009a000 -s .init.text
0xffffffffc00a4000 -s .text.unlikely 0xffffffffc009971d -s .exit.text 0xffffffffc00998ea
-s .rodata 0xffffffffc009a040 -s .rodata.str1.1 0xffffffffc009a4d0 -s .rodata.str1.8
0xffffffffc009ad98 -s .smp_locks 0xffffffffc009be14 -s .parainstructions
0xffffffffc009be98 -s __param 0xffffffffc009beb8 -s __mcount_loc 0xffffffffc009bf18 -s
.data 0xffffffffc009d000 -s .init.data 0xffffffffc00a5000 -s .gnu.linkonce.this_module
0xffffffffc009db00 -s .bss 0xffffffffc009dd40 -s .symtab 0xffffffffc00a6000 -s .strtab
0xffffffffc00a80b8
> add symbol table from file
"/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug"
at
> .text_addr = 0xffffffffc0092000
> .note.gnu.build-id_addr = 0xffffffffc009a000
> .init.text_addr = 0xffffffffc00a4000
> .text.unlikely_addr = 0xffffffffc009971d
> .exit.text_addr = 0xffffffffc00998ea
> .rodata_addr = 0xffffffffc009a040
> .rodata.str1.1_addr = 0xffffffffc009a4d0
> .rodata.str1.8_addr = 0xffffffffc009ad98
> .smp_locks_addr = 0xffffffffc009be14
> .parainstructions_addr = 0xffffffffc009be98
> __param_addr = 0xffffffffc009beb8
> __mcount_loc_addr = 0xffffffffc009bf18
> .data_addr = 0xffffffffc009d000
> .init.data_addr = 0xffffffffc00a5000
> .gnu.linkonce.this_module_addr = 0xffffffffc009db00
> .bss_addr = 0xffffffffc009dd40
> .symtab_addr = 0xffffffffc00a6000
> .strtab_addr = 0xffffffffc00a80b8
> warning: section .symtab not found in
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug
> warning: section .strtab not found in
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug
> MODULE NAME BASE SIZE OBJECT FILE
> ffffffffc009db00 floppy ffffffffc0092000 69417
/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug
>