[Crash-utility] Re: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

On 2023/11/08 12:01, HAGIO KAZUHITO(萩尾 一仁) wrote: Hmm, it looks like several sections are mapped outside of the current module range. What if they are not specified or they are mapped to e.g. an unused range..? or any idea? crash-dev> sym -m floppy | grep MODULE ffffffffc0092000 MODULE START: floppy ffffffffc00a2f29 MODULE END: floppy crash-dev> set debug 1 debug: 1 crash-dev> mod -s floppy ffffffffc009a000 .note.gnu.build-id ffffffffc0092000 .text ffffffffc00a4000 .init.text <<< higher than MODULE END ffffffffc009971d .text.unlikely ffffffffc00998ea .exit.text ffffffffc009a040 .rodata ffffffffc009a4d0 .rodata.str1.1 ffffffffc009ad98 .rodata.str1.8 ffffffffc009be14 .smp_locks ffffffffc009be98 .parainstructions ffffffffc009beb8 __param ffffffffc009bf18 __mcount_loc ffffffffc009d000 .data ffffffffc00a5000 .init.data <<< ffffffffc009db00 .gnu.linkonce.this_module ffffffffc009dd40 .bss ffffffffc00a6000 .symtab <<< ffffffffc00a80b8 .strtab <<< add-symbol-file /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug 0xffffffffc0092000 -s .note.gnu.build-id 0xffffffffc009a000 -s .init.text 0xffffffffc00a4000 -s .text.unlikely 0xffffffffc009971d -s .exit.text 0xffffffffc00998ea -s .rodata 0xffffffffc009a040 -s .rodata.str1.1 0xffffffffc009a4d0 -s .rodata.str1.8 0xffffffffc009ad98 -s .smp_locks 0xffffffffc009be14 -s .parainstructions 0xffffffffc009be98 -s __param 0xffffffffc009beb8 -s __mcount_loc 0xffffffffc009bf18 -s .data 0xffffffffc009d000 -s .init.data 0xffffffffc00a5000 -s .gnu.linkonce.this_module 0xffffffffc009db00 -s .bss 0xffffffffc009dd40 -s .symtab 0xffffffffc00a6000 -s .strtab 0xffffffffc00a80b8 add symbol table from file "/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug" at .text_addr = 0xffffffffc0092000 .note.gnu.build-id_addr = 0xffffffffc009a000 .init.text_addr = 0xffffffffc00a4000 .text.unlikely_addr = 0xffffffffc009971d .exit.text_addr = 0xffffffffc00998ea .rodata_addr = 0xffffffffc009a040 .rodata.str1.1_addr = 0xffffffffc009a4d0 .rodata.str1.8_addr = 0xffffffffc009ad98 .smp_locks_addr = 0xffffffffc009be14 .parainstructions_addr = 0xffffffffc009be98 __param_addr = 0xffffffffc009beb8 __mcount_loc_addr = 0xffffffffc009bf18 .data_addr = 0xffffffffc009d000 .init.data_addr = 0xffffffffc00a5000 .gnu.linkonce.this_module_addr = 0xffffffffc009db00 .bss_addr = 0xffffffffc009dd40 .symtab_addr = 0xffffffffc00a6000 .strtab_addr = 0xffffffffc00a80b8 warning: section .symtab not found in /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug warning: section .strtab not found in /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug MODULE NAME BASE SIZE OBJECT FILE ffffffffc009db00 floppy ffffffffc0092000 69417 /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug Thanks, Kazu > > Thanks, > Kazu > > > On 2023/11/07 17:20, Tao Liu wrote: >> Hi Kazu, >> >> On Tue, Nov 7, 2023 at 1:07 PM HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com&gt; wrote: >>> >>> Hi Tao, >>> >>> thank you for the patches. >>> >>> On 2023/11/02 21:09, Tao Liu wrote: >>>> The address range may overlap for kernel modules global block vector, >>>> for example: >>>> >>>> module compunit_symtab block start block end >>>> floppy floppy.c ffffffffc0092000 ffffffffc00a4fa6 >>>> virtio_blk virtio_blk.c ffffffffc00a4000 ffffffffc00fd080 >>> >>> hmm, this condition looks strange in the first place, is it inevitable? >>> In other words, I wonder if there might be something wrong with how to >>> load module symbol files. >>> >> >> Honestly I'm not sure why there is address overlap of the 2 modules >> global block, it shouldn't happen from my view. But it can happen on >> another kernel(4.18.0-425.13.1.el8_7.x86_64), see the results blow: >> >> module compunit_symtab block start block end >> libata.ko.debug libata-core.c ffffffffc054f000 ffffffffc0591b18 >> mgag200.ko.debug mgag200_drv.c ffffffffc03f3000 ffffffffc0595b0e >> >> module address boundary: >> crash> sym -l >> ffffffffc054f000 MODULE START: libata >> ...snip... >> ffffffffc0590000 MODULE END: libata >> ffffffffc0591000 MODULE START: mgag200 >> ...snip... >> ffffffffc059a000 MODULE END: mgag200 >> >>> What is the kernel version? >> >> The kernel which I used to create the patch is 3.10.0-693.2.2.el7.x86_64. >> >>> Do you have any patch to debug the gdb internal information? >> >> Sure, simple debug patch as follows: >> diff --git gdb-10.2/gdb/symtab.c.orig gdb-10.2/gdb/symtab.c >> index 82605cc..3af3570 100644 >> --- gdb-10.2/gdb/symtab.c.orig >> +++ gdb-10.2/gdb/symtab.c >> @@ -2939,6 +2939,8 @@ find_pc_sect_compunit_symtab (CORE_ADDR pc, >> struct obj_section *section) >> bv = COMPUNIT_BLOCKVECTOR (cust); >> b = BLOCKVECTOR_BLOCK (bv, GLOBAL_BLOCK); >> >> + printf("%s %s %lx %lx\n", objfile_filename(obj_file), >> cust->name, BLOCK_START (b), BLOCK_END (b)); >> + >> if (BLOCK_START (b) <= pc >> && BLOCK_END (b) > pc >> && (distance == 0 >>> >>> I'd like to check vmcores on hand.. >> >> I can share the vmcore with you by a private email later. >> >> Thanks, >> Tao Liu >> >>> >>> Thanks, >>> Kazu >>> >>>> >>>> According to the gdb source code, the distance(block end - block start) of >>>> floppy(0x12fa6) is smaller than virtio_blk(0x59080), so address >>>> 0xffffffffc00a45b0, 0xffffffffc00a4370, 0xffffffffc00a4260 are translated >>>> by floppy module instead of virtio_blk module. However according >>>> to crash, the address range for the 2 modules are: >>>> >>>> crash> sym -l >>>> ffffffffc0092000 MODULE START: floppy >>>> ...snip... >>>> ffffffffc00a2f29 MODULE END: floppy >>>> ffffffffc00a4000 MODULE START: virtio_blk >>>> ...snip... >>>> ffffffffc00a86ec MODULE END: virtio_blk >>>> >>>> So the block vector address arrange and distance is not a reliable way to >>>> determine the symbol belonging of a specific kernel module. This patch will >>>> let gdb to query the module address range from crash, in order to locate the >>>> symbol belongings more percisely. >>>> >>>> Without the patch: >>>> crash> mod -S >>>> crash> struct blk_mq_ops 0xffffffffc00a7160 >>>> struct blk_mq_ops { >>>> queue_rq = 0xffffffffc00a45b0 <floppy_module_init+1151>, <-- symbol translated from module floppy >>>> map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, >>>> ...snip... >>>> complete = 0xffffffffc00a4370 <floppy_module_init+575>, >>>> init_request = 0xffffffffc00a4260 <floppy_module_init+303>, >>>> ...snip... >>>> } >>>> >>>> With the patch: >>>> crash> mod -S >>>> crash> struct blk_mq_ops 0xffffffffc00a7160 >>>> struct blk_mq_ops { >>>> queue_rq = 0xffffffffc00a45b0 <virtio_queue_rq>, <-- symbol translated from module virtio_blk >>>> map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, >>>> ...snip... >>>> complete = 0xffffffffc00a4370 <virtblk_request_done>, >>>> init_request = 0xffffffffc00a4260 <virtblk_init_request>, >>>> ...snip... >>>> } >>>> >>>> Signed-off-by: Tao Liu <ltao(a)redhat.com&gt; >>>> --- >>>> defs.h | 2 ++ >>>> gdb-10.2.patch | 37 ++++++++++++++++++++++++++++++++++++- >>>> gdb_interface.c | 18 ++++++++++++++++++ >>>> 3 files changed, 56 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/defs.h b/defs.h >>>> index 788f63a..09e6738 100644 >>>> --- a/defs.h >>>> +++ b/defs.h >>>> @@ -5130,6 +5130,7 @@ struct syment *symbol_search(char *); >>>> int gdb_line_number_callback(ulong, ulong, ulong); >>>> int gdb_print_callback(ulong); >>>> char *gdb_lookup_module_symbol(ulong, ulong *); >>>> +ulong gdb_get_module_boundary(const char *, bool); >>>> extern "C" int same_file(char *, char *); >>>> #endif >>>> >>>> @@ -7687,6 +7688,7 @@ int gdb_readmem_callback(ulong, void *, int, int); >>>> int gdb_line_number_callback(ulong, ulong, ulong); >>>> int gdb_print_callback(ulong); >>>> char *gdb_lookup_module_symbol(ulong, ulong *); >>>> +ulong gdb_get_module_boundary(const char *, bool); >>>> void gdb_error_hook(void); >>>> void restore_gdb_sanity(void); >>>> int is_gdb_command(int, ulong); >>>> diff --git a/gdb-10.2.patch b/gdb-10.2.patch >>>> index 31135ca..a5c10d3 100644 >>>> --- a/gdb-10.2.patch >>>> +++ b/gdb-10.2.patch >>>> @@ -3203,4 +3203,39 @@ exit 0 >>>> +#endif >>>> for (compunit_symtab *cust : obj_file->compunits ()) >>>> { >>>> - const struct block *b; >>>> \ No newline at end of file >>>> + const struct block *b; >>>> +--- gdb-10.2/gdb/symtab.c.orig >>>> ++++ gdb-10.2/gdb/symtab.c >>>> +@@ -2895,6 +2895,10 @@ iterate_over_symbols_terminated >>>> + return callback (&block_sym); >>>> + } >>>> + >>>> ++#ifdef CRASH_MERGE >>>> ++extern "C" ulong gdb_get_module_boundary(const char *, bool); >>>> ++#endif >>>> ++ >>>> + /* Find the compunit symtab associated with PC and SECTION. >>>> + This will read in debug info as necessary. */ >>>> + >>>> +@@ -2932,11 +2936,21 @@ find_pc_sect_compunit_symtab (CORE_ADDR pc, struct obj_section *section) >>>> + for (objfile *obj_file : current_program_space->objfiles ()) >>>> + { >>>> + #ifdef CRASH_MERGE >>>> ++ size_t sep; >>>> + std::string objfile_name = objfile_filename(obj_file); >>>> ++ const char *mod_name; >>>> + >>>> + if (objfile_name.find(".ko") != std::string::npos) { >>>> + if (obj_file->sf && obj_file->compunit_symtabs == nullptr) >>>> + obj_file->sf->qf->expand_all_symtabs(obj_file); >>>> ++ sep = objfile_name.find_last_of("/"); >>>> ++ objfile_name = objfile_name.substr(sep + 1, objfile_name.size() - sep - 1); >>>> ++ sep = objfile_name.find_first_of("."); >>>> ++ objfile_name = objfile_name.substr(0, sep); >>>> ++ mod_name = objfile_name.c_str(); >>>> ++ if (pc < gdb_get_module_boundary(mod_name, true) || >>>> ++ pc > gdb_get_module_boundary(mod_name, false)) >>>> ++ continue; >>>> + } >>>> + #endif >>>> + for (compunit_symtab *cust : obj_file->compunits ()) >>>> diff --git a/gdb_interface.c b/gdb_interface.c >>>> index b14319c..a58fd68 100644 >>>> --- a/gdb_interface.c >>>> +++ b/gdb_interface.c >>>> @@ -947,6 +947,24 @@ gdb_lookup_module_symbol(ulong addr, ulong *offset) >>>> } >>>> } >>>> >>>> +ulong >>>> +gdb_get_module_boundary(const char *module, bool is_start) >>>> +{ >>>> + char buf[256]; >>>> + struct syment *sp; >>>> + >>>> + if (is_start) { >>>> + snprintf(buf, sizeof(buf), "_MODULE_START_%s", module); >>>> + } else { >>>> + snprintf(buf, sizeof(buf), "_MODULE_END_%s", module); >>>> + } >>>> + sp = symbol_search(buf); >>>> + if (sp) >>>> + return sp->value; >>>> + else >>>> + return is_start ? 0 : (ulong)(-1); >>>> +} >>>> + >>>> /* >>>> * Used by gdb_interface() to catch gdb-related errors, if desired. >>>> */ > -- > Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io > To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > Contribution Guidelines: https://github.com/crash-utility/crash/wiki