[PATCH 1/2] symbols: expand kernel modules symtable before symbols translation

[PATCH v2] symbols: skip load...

[PATCH] Fix printing incorrect...

Tao Liu

Thursday, 2 November 2023 Thu, 2 Nov '23

7:09 a.m.

The kernel modules symbol translation may change after a c expression evaluation. without patch: crash> mod -S crash> struct blk_mq_ops 0xffffffffc00a7160 struct blk_mq_ops { queue_rq = 0xffffffffc00a45b0 <virtio_queue_rq>, <--symbol translated from kernel map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, ...snip... complete = 0xffffffffc00a4370 <virtblk_request_done>, init_request = 0xffffffffc00a4260 <virtblk_init_request>, ...snip... } crash> px ((struct request *)0xffff880fdb246000)->q->mq_ops $1 = (struct blk_mq_ops *) 0xffffffffc00a7160 <virtio_mq_ops> crash> struct blk_mq_ops 0xffffffffc00a7160 struct blk_mq_ops { queue_rq = 0xffffffffc00a45b0 <floppy_module_init+1151>, <--symbol translated from module map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, ...snip... complete = 0xffffffffc00a4370 <floppy_module_init+575>, init_request = 0xffffffffc00a4260 <floppy_module_init+303>, ...snip... } with patch: crash> mod -S crash> struct blk_mq_ops 0xffffffffc00a7160 struct blk_mq_ops { queue_rq = 0xffffffffc00a45b0 <floppy_module_init+1151>, <--symbol translated from module map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, ...snip... complete = 0xffffffffc00a4370 <floppy_module_init+575>, init_request = 0xffffffffc00a4260 <floppy_module_init+303>, ..snip... } crash> px ((struct request *)0xffff880fdb246000)->q->mq_ops $1 = (struct blk_mq_ops *) 0xffffffffc00a7160 <virtio_mq_ops> crash> struct blk_mq_ops 0xffffffffc00a7160 struct blk_mq_ops { queue_rq = 0xffffffffc00a45b0 <floppy_module_init+1151>, <--symbol translated from module map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, ...snip... complete = 0xffffffffc00a4370 <floppy_module_init+575>, init_request = 0xffffffffc00a4260 <floppy_module_init+303>, ...snip... } The root cause for the changing of symbol translation is, after "mod -S", the kernel modules files "*.ko.debug" will be loaded. However the compile unit symtable of the kernel modules may not get expanded. As a result, the symtable of kernel modules, or obj_file->compunit_symtabs is nullptr, which don't take any effect for gdb symbol translation, it is unexpected. A c expression evaluation will trigger such an expansion. This patch will make sure symtable always get expanded before gdb symbol translation. Signed-off-by: Tao Liu <ltao(a)redhat.com> --- gdb-10.2.patch | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/gdb-10.2.patch b/gdb-10.2.patch index d81030d..31135ca 100644 --- a/gdb-10.2.patch +++ b/gdb-10.2.patch @@ -3187,3 +3187,20 @@ exit 0 result = stringtab + symbol_entry->_n._n_n._n_offset; } else +--- gdb-10.2/gdb/symtab.c.orig ++++ gdb-10.2/gdb/symtab.c +@@ -2931,6 +2931,14 @@ find_pc_sect_compunit_symtab (CORE_ADDR pc, struct obj_section *section) + + for (objfile *obj_file : current_program_space->objfiles ()) + { ++#ifdef CRASH_MERGE ++ std::string objfile_name = objfile_filename(obj_file); ++ ++ if (objfile_name.find(".ko") != std::string::npos) { ++ if (obj_file->sf && obj_file->compunit_symtabs == nullptr) ++ obj_file->sf->qf->expand_all_symtabs(obj_file); ++ } ++#endif + for (compunit_symtab *cust : obj_file->compunits ()) + { + const struct block *b; \ No newline at end of file -- 2.40.1

Show replies by date

Tao Liu

Thursday, 2 November Thu, 2 Nov

7:09 a.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

The address range may overlap for kernel modules global block vector, for example: module compunit_symtab block start block end floppy floppy.c ffffffffc0092000 ffffffffc00a4fa6 virtio_blk virtio_blk.c ffffffffc00a4000 ffffffffc00fd080 According to the gdb source code, the distance(block end - block start) of floppy(0x12fa6) is smaller than virtio_blk(0x59080), so address 0xffffffffc00a45b0, 0xffffffffc00a4370, 0xffffffffc00a4260 are translated by floppy module instead of virtio_blk module. However according to crash, the address range for the 2 modules are: crash> sym -l ffffffffc0092000 MODULE START: floppy ...snip... ffffffffc00a2f29 MODULE END: floppy ffffffffc00a4000 MODULE START: virtio_blk ...snip... ffffffffc00a86ec MODULE END: virtio_blk So the block vector address arrange and distance is not a reliable way to determine the symbol belonging of a specific kernel module. This patch will let gdb to query the module address range from crash, in order to locate the symbol belongings more percisely. Without the patch: crash> mod -S crash> struct blk_mq_ops 0xffffffffc00a7160 struct blk_mq_ops { queue_rq = 0xffffffffc00a45b0 <floppy_module_init+1151>, <-- symbol translated from module floppy map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, ...snip... complete = 0xffffffffc00a4370 <floppy_module_init+575>, init_request = 0xffffffffc00a4260 <floppy_module_init+303>, ...snip... } With the patch: crash> mod -S crash> struct blk_mq_ops 0xffffffffc00a7160 struct blk_mq_ops { queue_rq = 0xffffffffc00a45b0 <virtio_queue_rq>, <-- symbol translated from module virtio_blk map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, ...snip... complete = 0xffffffffc00a4370 <virtblk_request_done>, init_request = 0xffffffffc00a4260 <virtblk_init_request>, ...snip... } Signed-off-by: Tao Liu <ltao(a)redhat.com> --- defs.h | 2 ++ gdb-10.2.patch | 37 ++++++++++++++++++++++++++++++++++++- gdb_interface.c | 18 ++++++++++++++++++ 3 files changed, 56 insertions(+), 1 deletion(-) diff --git a/defs.h b/defs.h index 788f63a..09e6738 100644 --- a/defs.h +++ b/defs.h @@ -5130,6 +5130,7 @@ struct syment *symbol_search(char *); int gdb_line_number_callback(ulong, ulong, ulong); int gdb_print_callback(ulong); char *gdb_lookup_module_symbol(ulong, ulong *); +ulong gdb_get_module_boundary(const char *, bool); extern "C" int same_file(char *, char *); #endif @@ -7687,6 +7688,7 @@ int gdb_readmem_callback(ulong, void *, int, int); int gdb_line_number_callback(ulong, ulong, ulong); int gdb_print_callback(ulong); char *gdb_lookup_module_symbol(ulong, ulong *); +ulong gdb_get_module_boundary(const char *, bool); void gdb_error_hook(void); void restore_gdb_sanity(void); int is_gdb_command(int, ulong); diff --git a/gdb-10.2.patch b/gdb-10.2.patch index 31135ca..a5c10d3 100644 --- a/gdb-10.2.patch +++ b/gdb-10.2.patch @@ -3203,4 +3203,39 @@ exit 0 +#endif for (compunit_symtab *cust : obj_file->compunits ()) { - const struct block *b; \ No newline at end of file + const struct block *b; +--- gdb-10.2/gdb/symtab.c.orig ++++ gdb-10.2/gdb/symtab.c +@@ -2895,6 +2895,10 @@ iterate_over_symbols_terminated + return callback (&block_sym); + } + ++#ifdef CRASH_MERGE ++extern "C" ulong gdb_get_module_boundary(const char *, bool); ++#endif ++ + /* Find the compunit symtab associated with PC and SECTION. + This will read in debug info as necessary. */ + +@@ -2932,11 +2936,21 @@ find_pc_sect_compunit_symtab (CORE_ADDR pc, struct obj_section *section) + for (objfile *obj_file : current_program_space->objfiles ()) + { + #ifdef CRASH_MERGE ++ size_t sep; + std::string objfile_name = objfile_filename(obj_file); ++ const char *mod_name; + + if (objfile_name.find(".ko") != std::string::npos) { + if (obj_file->sf && obj_file->compunit_symtabs == nullptr) + obj_file->sf->qf->expand_all_symtabs(obj_file); ++ sep = objfile_name.find_last_of("/"); ++ objfile_name = objfile_name.substr(sep + 1, objfile_name.size() - sep - 1); ++ sep = objfile_name.find_first_of("."); ++ objfile_name = objfile_name.substr(0, sep); ++ mod_name = objfile_name.c_str(); ++ if (pc < gdb_get_module_boundary(mod_name, true) || ++ pc > gdb_get_module_boundary(mod_name, false)) ++ continue; + } + #endif + for (compunit_symtab *cust : obj_file->compunits ()) diff --git a/gdb_interface.c b/gdb_interface.c index b14319c..a58fd68 100644 --- a/gdb_interface.c +++ b/gdb_interface.c @@ -947,6 +947,24 @@ gdb_lookup_module_symbol(ulong addr, ulong *offset) } } +ulong +gdb_get_module_boundary(const char *module, bool is_start) +{ + char buf[256]; + struct syment *sp; + + if (is_start) { + snprintf(buf, sizeof(buf), "_MODULE_START_%s", module); + } else { + snprintf(buf, sizeof(buf), "_MODULE_END_%s", module); + } + sp = symbol_search(buf); + if (sp) + return sp->value; + else + return is_start ? 0 : (ulong)(-1); +} + /* * Used by gdb_interface() to catch gdb-related errors, if desired. */ -- 2.40.1

HAGIO KAZUHITO(萩尾　一仁)

Monday, 6 November Mon, 6 Nov

11:06 p.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

Hi Tao, thank you for the patches. On 2023/11/02 21:09, Tao Liu wrote:

...

hmm, this condition looks strange in the first place, is it inevitable? In other words, I wonder if there might be something wrong with how to load module symbol files. What is the kernel version? Do you have any patch to debug the gdb internal information? I'd like to check vmcores on hand.. Thanks, Kazu > > According to the gdb source code, the distance(block end - block start) of > floppy(0x12fa6) is smaller than virtio_blk(0x59080), so address > 0xffffffffc00a45b0, 0xffffffffc00a4370, 0xffffffffc00a4260 are translated > by floppy module instead of virtio_blk module. However according > to crash, the address range for the 2 modules are: > > crash> sym -l > ffffffffc0092000 MODULE START: floppy > ...snip... > ffffffffc00a2f29 MODULE END: floppy > ffffffffc00a4000 MODULE START: virtio_blk > ...snip... > ffffffffc00a86ec MODULE END: virtio_blk > > So the block vector address arrange and distance is not a reliable way to > determine the symbol belonging of a specific kernel module. This patch will > let gdb to query the module address range from crash, in order to locate the > symbol belongings more percisely. > > Without the patch: > crash> mod -S > crash> struct blk_mq_ops 0xffffffffc00a7160 > struct blk_mq_ops { > queue_rq = 0xffffffffc00a45b0 <floppy_module_init+1151>, <-- symbol translated from module floppy > map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, > ...snip... > complete = 0xffffffffc00a4370 <floppy_module_init+575>, > init_request = 0xffffffffc00a4260 <floppy_module_init+303>, > ...snip... > } > > With the patch: > crash> mod -S > crash> struct blk_mq_ops 0xffffffffc00a7160 > struct blk_mq_ops { > queue_rq = 0xffffffffc00a45b0 <virtio_queue_rq>, <-- symbol translated from module virtio_blk > map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, > ...snip... > complete = 0xffffffffc00a4370 <virtblk_request_done>, > init_request = 0xffffffffc00a4260 <virtblk_init_request>, > ...snip... > } > > Signed-off-by: Tao Liu <ltao(a)redhat.com> > --- > defs.h | 2 ++ > gdb-10.2.patch | 37 ++++++++++++++++++++++++++++++++++++- > gdb_interface.c | 18 ++++++++++++++++++ > 3 files changed, 56 insertions(+), 1 deletion(-) > > diff --git a/defs.h b/defs.h > index 788f63a..09e6738 100644 > --- a/defs.h > +++ b/defs.h > @@ -5130,6 +5130,7 @@ struct syment *symbol_search(char *); > int gdb_line_number_callback(ulong, ulong, ulong); > int gdb_print_callback(ulong); > char *gdb_lookup_module_symbol(ulong, ulong *); > +ulong gdb_get_module_boundary(const char *, bool); > extern "C" int same_file(char *, char *); > #endif > > @@ -7687,6 +7688,7 @@ int gdb_readmem_callback(ulong, void *, int, int); > int gdb_line_number_callback(ulong, ulong, ulong); > int gdb_print_callback(ulong); > char *gdb_lookup_module_symbol(ulong, ulong *); > +ulong gdb_get_module_boundary(const char *, bool); > void gdb_error_hook(void); > void restore_gdb_sanity(void); > int is_gdb_command(int, ulong); > diff --git a/gdb-10.2.patch b/gdb-10.2.patch > index 31135ca..a5c10d3 100644 > --- a/gdb-10.2.patch > +++ b/gdb-10.2.patch > @@ -3203,4 +3203,39 @@ exit 0 > +#endif > for (compunit_symtab *cust : obj_file->compunits ()) > { > - const struct block *b; > \ No newline at end of file > + const struct block *b; > +--- gdb-10.2/gdb/symtab.c.orig > ++++ gdb-10.2/gdb/symtab.c > +@@ -2895,6 +2895,10 @@ iterate_over_symbols_terminated > + return callback (&block_sym); > + } > + > ++#ifdef CRASH_MERGE > ++extern "C" ulong gdb_get_module_boundary(const char *, bool); > ++#endif > ++ > + /* Find the compunit symtab associated with PC and SECTION. > + This will read in debug info as necessary. */ > + > +@@ -2932,11 +2936,21 @@ find_pc_sect_compunit_symtab (CORE_ADDR pc, struct obj_section *section) > + for (objfile *obj_file : current_program_space->objfiles ()) > + { > + #ifdef CRASH_MERGE > ++ size_t sep; > + std::string objfile_name = objfile_filename(obj_file); > ++ const char *mod_name; > + > + if (objfile_name.find(".ko") != std::string::npos) { > + if (obj_file->sf && obj_file->compunit_symtabs == nullptr) > + obj_file->sf->qf->expand_all_symtabs(obj_file); > ++ sep = objfile_name.find_last_of("/"); > ++ objfile_name = objfile_name.substr(sep + 1, objfile_name.size() - sep - 1); > ++ sep = objfile_name.find_first_of("."); > ++ objfile_name = objfile_name.substr(0, sep); > ++ mod_name = objfile_name.c_str(); > ++ if (pc < gdb_get_module_boundary(mod_name, true) || > ++ pc > gdb_get_module_boundary(mod_name, false)) > ++ continue; > + } > + #endif > + for (compunit_symtab *cust : obj_file->compunits ()) > diff --git a/gdb_interface.c b/gdb_interface.c > index b14319c..a58fd68 100644 > --- a/gdb_interface.c > +++ b/gdb_interface.c > @@ -947,6 +947,24 @@ gdb_lookup_module_symbol(ulong addr, ulong *offset) > } > } > > +ulong > +gdb_get_module_boundary(const char *module, bool is_start) > +{ > + char buf[256]; > + struct syment *sp; > + > + if (is_start) { > + snprintf(buf, sizeof(buf), "_MODULE_START_%s", module); > + } else { > + snprintf(buf, sizeof(buf), "_MODULE_END_%s", module); > + } > + sp = symbol_search(buf); > + if (sp) > + return sp->value; > + else > + return is_start ? 0 : (ulong)(-1); > +} > + > /* > * Used by gdb_interface() to catch gdb-related errors, if desired. > */

Tao Liu

Tuesday, 7 November Tue, 7 Nov

2:20 a.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

Hi Kazu, On Tue, Nov 7, 2023 at 1:07 PM HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com> wrote:

...

Hi Tao, thank you for the patches. On 2023/11/02 21:09, Tao Liu wrote: > The address range may overlap for kernel modules global block vector, > for example: > > module compunit_symtab block start block end > floppy floppy.c ffffffffc0092000 ffffffffc00a4fa6 > virtio_blk virtio_blk.c ffffffffc00a4000 ffffffffc00fd080 hmm, this condition looks strange in the first place, is it inevitable? In other words, I wonder if there might be something wrong with how to load module symbol files.

Honestly I'm not sure why there is address overlap of the 2 modules global block, it shouldn't happen from my view. But it can happen on another kernel(4.18.0-425.13.1.el8_7.x86_64), see the results blow: module compunit_symtab block start block end libata.ko.debug libata-core.c ffffffffc054f000 ffffffffc0591b18 mgag200.ko.debug mgag200_drv.c ffffffffc03f3000 ffffffffc0595b0e module address boundary: crash> sym -l ffffffffc054f000 MODULE START: libata ...snip... ffffffffc0590000 MODULE END: libata ffffffffc0591000 MODULE START: mgag200 ...snip... ffffffffc059a000 MODULE END: mgag200

...

What is the kernel version?

The kernel which I used to create the patch is 3.10.0-693.2.2.el7.x86_64.

...

Do you have any patch to debug the gdb internal information?

Sure, simple debug patch as follows: diff --git gdb-10.2/gdb/symtab.c.orig gdb-10.2/gdb/symtab.c index 82605cc..3af3570 100644 --- gdb-10.2/gdb/symtab.c.orig +++ gdb-10.2/gdb/symtab.c @@ -2939,6 +2939,8 @@ find_pc_sect_compunit_symtab (CORE_ADDR pc, struct obj_section *section) bv = COMPUNIT_BLOCKVECTOR (cust); b = BLOCKVECTOR_BLOCK (bv, GLOBAL_BLOCK); + printf("%s %s %lx %lx\n", objfile_filename(obj_file), cust->name, BLOCK_START (b), BLOCK_END (b)); + if (BLOCK_START (b) <= pc && BLOCK_END (b) > pc && (distance == 0

...

I'd like to check vmcores on hand..

I can share the vmcore with you by a private email later. Thanks, Tao Liu

...

Thanks, Kazu > > According to the gdb source code, the distance(block end - block start) of > floppy(0x12fa6) is smaller than virtio_blk(0x59080), so address > 0xffffffffc00a45b0, 0xffffffffc00a4370, 0xffffffffc00a4260 are translated > by floppy module instead of virtio_blk module. However according > to crash, the address range for the 2 modules are: > > crash> sym -l > ffffffffc0092000 MODULE START: floppy > ...snip... > ffffffffc00a2f29 MODULE END: floppy > ffffffffc00a4000 MODULE START: virtio_blk > ...snip... > ffffffffc00a86ec MODULE END: virtio_blk > > So the block vector address arrange and distance is not a reliable way to > determine the symbol belonging of a specific kernel module. This patch will > let gdb to query the module address range from crash, in order to locate the > symbol belongings more percisely. > > Without the patch: > crash> mod -S > crash> struct blk_mq_ops 0xffffffffc00a7160 > struct blk_mq_ops { > queue_rq = 0xffffffffc00a45b0 <floppy_module_init+1151>, <-- symbol translated from module floppy > map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, > ...snip... > complete = 0xffffffffc00a4370 <floppy_module_init+575>, > init_request = 0xffffffffc00a4260 <floppy_module_init+303>, > ...snip... > } > > With the patch: > crash> mod -S > crash> struct blk_mq_ops 0xffffffffc00a7160 > struct blk_mq_ops { > queue_rq = 0xffffffffc00a45b0 <virtio_queue_rq>, <-- symbol translated from module virtio_blk > map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, > ...snip... > complete = 0xffffffffc00a4370 <virtblk_request_done>, > init_request = 0xffffffffc00a4260 <virtblk_init_request>, > ...snip... > } > > Signed-off-by: Tao Liu <ltao(a)redhat.com> > --- > defs.h | 2 ++ > gdb-10.2.patch | 37 ++++++++++++++++++++++++++++++++++++- > gdb_interface.c | 18 ++++++++++++++++++ > 3 files changed, 56 insertions(+), 1 deletion(-) > > diff --git a/defs.h b/defs.h > index 788f63a..09e6738 100644 > --- a/defs.h > +++ b/defs.h > @@ -5130,6 +5130,7 @@ struct syment *symbol_search(char *); > int gdb_line_number_callback(ulong, ulong, ulong); > int gdb_print_callback(ulong); > char *gdb_lookup_module_symbol(ulong, ulong *); > +ulong gdb_get_module_boundary(const char *, bool); > extern "C" int same_file(char *, char *); > #endif > > @@ -7687,6 +7688,7 @@ int gdb_readmem_callback(ulong, void *, int, int); > int gdb_line_number_callback(ulong, ulong, ulong); > int gdb_print_callback(ulong); > char *gdb_lookup_module_symbol(ulong, ulong *); > +ulong gdb_get_module_boundary(const char *, bool); > void gdb_error_hook(void); > void restore_gdb_sanity(void); > int is_gdb_command(int, ulong); > diff --git a/gdb-10.2.patch b/gdb-10.2.patch > index 31135ca..a5c10d3 100644 > --- a/gdb-10.2.patch > +++ b/gdb-10.2.patch > @@ -3203,4 +3203,39 @@ exit 0 > +#endif > for (compunit_symtab *cust : obj_file->compunits ()) > { > - const struct block *b; > \ No newline at end of file > + const struct block *b; > +--- gdb-10.2/gdb/symtab.c.orig > ++++ gdb-10.2/gdb/symtab.c > +@@ -2895,6 +2895,10 @@ iterate_over_symbols_terminated > + return callback (&block_sym); > + } > + > ++#ifdef CRASH_MERGE > ++extern "C" ulong gdb_get_module_boundary(const char *, bool); > ++#endif > ++ > + /* Find the compunit symtab associated with PC and SECTION. > + This will read in debug info as necessary. */ > + > +@@ -2932,11 +2936,21 @@ find_pc_sect_compunit_symtab (CORE_ADDR pc, struct obj_section *section) > + for (objfile *obj_file : current_program_space->objfiles ()) > + { > + #ifdef CRASH_MERGE > ++ size_t sep; > + std::string objfile_name = objfile_filename(obj_file); > ++ const char *mod_name; > + > + if (objfile_name.find(".ko") != std::string::npos) { > + if (obj_file->sf && obj_file->compunit_symtabs == nullptr) > + obj_file->sf->qf->expand_all_symtabs(obj_file); > ++ sep = objfile_name.find_last_of("/"); > ++ objfile_name = objfile_name.substr(sep + 1, objfile_name.size() - sep - 1); > ++ sep = objfile_name.find_first_of("."); > ++ objfile_name = objfile_name.substr(0, sep); > ++ mod_name = objfile_name.c_str(); > ++ if (pc < gdb_get_module_boundary(mod_name, true) || > ++ pc > gdb_get_module_boundary(mod_name, false)) > ++ continue; > + } > + #endif > + for (compunit_symtab *cust : obj_file->compunits ()) > diff --git a/gdb_interface.c b/gdb_interface.c > index b14319c..a58fd68 100644 > --- a/gdb_interface.c > +++ b/gdb_interface.c > @@ -947,6 +947,24 @@ gdb_lookup_module_symbol(ulong addr, ulong *offset) > } > } > > +ulong > +gdb_get_module_boundary(const char *module, bool is_start) > +{ > + char buf[256]; > + struct syment *sp; > + > + if (is_start) { > + snprintf(buf, sizeof(buf), "_MODULE_START_%s", module); > + } else { > + snprintf(buf, sizeof(buf), "_MODULE_END_%s", module); > + } > + sp = symbol_search(buf); > + if (sp) > + return sp->value; > + else > + return is_start ? 0 : (ulong)(-1); > +} > + > /* > * Used by gdb_interface() to catch gdb-related errors, if desired. > */

HAGIO KAZUHITO(萩尾　一仁)

9:01 p.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

Hi Tao, thank you for the information. I'm looking into it, I noticed that the unexpected symbol "floopy_module_init" is in section .init.text. Crash side doesn't have the symbol info, probably the address is already freed and reused by the virtio_blk module? crash> mod -S ... crash> sym -m floppy | grep MODULE ffffffffc0092000 MODULE START: floppy ffffffffc00a2f29 MODULE END: floppy crash> sym -m virtio_blk | grep MODULE ffffffffc00a4000 MODULE START: virtio_blk ffffffffc00a86ec MODULE END: virtio_blk crash> gdb info symbol floppy_module_init init_module in section .init.text of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug virtblk_freeze + 33 in section .text of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/virtio_blk.ko.debug crash> gdb info address floppy_module_init Symbol "floppy_module_init" is a function at address 0xffffffffc00a4131. crash> sym floppy_module_init symbol not found: floppy_module_init possible alternatives: (none found) So still not sure, but this might be an issue of handling the init section/symbols of modules.. Thanks, Kazu On 2023/11/07 17:20, Tao Liu wrote: > Hi Kazu, > > On Tue, Nov 7, 2023 at 1:07 PM HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com> wrote: >> >> Hi Tao, >> >> thank you for the patches. >> >> On 2023/11/02 21:09, Tao Liu wrote: >>> The address range may overlap for kernel modules global block vector, >>> for example: >>> >>> module compunit_symtab block start block end >>> floppy floppy.c ffffffffc0092000 ffffffffc00a4fa6 >>> virtio_blk virtio_blk.c ffffffffc00a4000 ffffffffc00fd080 >> >> hmm, this condition looks strange in the first place, is it inevitable? >> In other words, I wonder if there might be something wrong with how to >> load module symbol files. >> > > Honestly I'm not sure why there is address overlap of the 2 modules > global block, it shouldn't happen from my view. But it can happen on > another kernel(4.18.0-425.13.1.el8_7.x86_64), see the results blow: > > module compunit_symtab block start block end > libata.ko.debug libata-core.c ffffffffc054f000 ffffffffc0591b18 > mgag200.ko.debug mgag200_drv.c ffffffffc03f3000 ffffffffc0595b0e > > module address boundary: > crash> sym -l > ffffffffc054f000 MODULE START: libata > ...snip... > ffffffffc0590000 MODULE END: libata > ffffffffc0591000 MODULE START: mgag200 > ...snip... > ffffffffc059a000 MODULE END: mgag200 > >> What is the kernel version? > > The kernel which I used to create the patch is 3.10.0-693.2.2.el7.x86_64. > >> Do you have any patch to debug the gdb internal information? > > Sure, simple debug patch as follows: > diff --git gdb-10.2/gdb/symtab.c.orig gdb-10.2/gdb/symtab.c > index 82605cc..3af3570 100644 > --- gdb-10.2/gdb/symtab.c.orig > +++ gdb-10.2/gdb/symtab.c > @@ -2939,6 +2939,8 @@ find_pc_sect_compunit_symtab (CORE_ADDR pc, > struct obj_section *section) > bv = COMPUNIT_BLOCKVECTOR (cust); > b = BLOCKVECTOR_BLOCK (bv, GLOBAL_BLOCK); > > + printf("%s %s %lx %lx\n", objfile_filename(obj_file), > cust->name, BLOCK_START (b), BLOCK_END (b)); > + > if (BLOCK_START (b) <= pc > && BLOCK_END (b) > pc > && (distance == 0 >> >> I'd like to check vmcores on hand.. > > I can share the vmcore with you by a private email later. > > Thanks, > Tao Liu > >> >> Thanks, >> Kazu >> >>> >>> According to the gdb source code, the distance(block end - block start) of >>> floppy(0x12fa6) is smaller than virtio_blk(0x59080), so address >>> 0xffffffffc00a45b0, 0xffffffffc00a4370, 0xffffffffc00a4260 are translated >>> by floppy module instead of virtio_blk module. However according >>> to crash, the address range for the 2 modules are: >>> >>> crash> sym -l >>> ffffffffc0092000 MODULE START: floppy >>> ...snip... >>> ffffffffc00a2f29 MODULE END: floppy >>> ffffffffc00a4000 MODULE START: virtio_blk >>> ...snip... >>> ffffffffc00a86ec MODULE END: virtio_blk >>> >>> So the block vector address arrange and distance is not a reliable way to >>> determine the symbol belonging of a specific kernel module. This patch will >>> let gdb to query the module address range from crash, in order to locate the >>> symbol belongings more percisely. >>> >>> Without the patch: >>> crash> mod -S >>> crash> struct blk_mq_ops 0xffffffffc00a7160 >>> struct blk_mq_ops { >>> queue_rq = 0xffffffffc00a45b0 <floppy_module_init+1151>, <-- symbol translated from module floppy >>> map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, >>> ...snip... >>> complete = 0xffffffffc00a4370 <floppy_module_init+575>, >>> init_request = 0xffffffffc00a4260 <floppy_module_init+303>, >>> ...snip... >>> } >>> >>> With the patch: >>> crash> mod -S >>> crash> struct blk_mq_ops 0xffffffffc00a7160 >>> struct blk_mq_ops { >>> queue_rq = 0xffffffffc00a45b0 <virtio_queue_rq>, <-- symbol translated from module virtio_blk >>> map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, >>> ...snip... >>> complete = 0xffffffffc00a4370 <virtblk_request_done>, >>> init_request = 0xffffffffc00a4260 <virtblk_init_request>, >>> ...snip... >>> } >>> >>> Signed-off-by: Tao Liu <ltao(a)redhat.com> >>> --- >>> defs.h | 2 ++ >>> gdb-10.2.patch | 37 ++++++++++++++++++++++++++++++++++++- >>> gdb_interface.c | 18 ++++++++++++++++++ >>> 3 files changed, 56 insertions(+), 1 deletion(-) >>> >>> diff --git a/defs.h b/defs.h >>> index 788f63a..09e6738 100644 >>> --- a/defs.h >>> +++ b/defs.h >>> @@ -5130,6 +5130,7 @@ struct syment *symbol_search(char *); >>> int gdb_line_number_callback(ulong, ulong, ulong); >>> int gdb_print_callback(ulong); >>> char *gdb_lookup_module_symbol(ulong, ulong *); >>> +ulong gdb_get_module_boundary(const char *, bool); >>> extern "C" int same_file(char *, char *); >>> #endif >>> >>> @@ -7687,6 +7688,7 @@ int gdb_readmem_callback(ulong, void *, int, int); >>> int gdb_line_number_callback(ulong, ulong, ulong); >>> int gdb_print_callback(ulong); >>> char *gdb_lookup_module_symbol(ulong, ulong *); >>> +ulong gdb_get_module_boundary(const char *, bool); >>> void gdb_error_hook(void); >>> void restore_gdb_sanity(void); >>> int is_gdb_command(int, ulong); >>> diff --git a/gdb-10.2.patch b/gdb-10.2.patch >>> index 31135ca..a5c10d3 100644 >>> --- a/gdb-10.2.patch >>> +++ b/gdb-10.2.patch >>> @@ -3203,4 +3203,39 @@ exit 0 >>> +#endif >>> for (compunit_symtab *cust : obj_file->compunits ()) >>> { >>> - const struct block *b; >>> \ No newline at end of file >>> + const struct block *b; >>> +--- gdb-10.2/gdb/symtab.c.orig >>> ++++ gdb-10.2/gdb/symtab.c >>> +@@ -2895,6 +2895,10 @@ iterate_over_symbols_terminated >>> + return callback (&block_sym); >>> + } >>> + >>> ++#ifdef CRASH_MERGE >>> ++extern "C" ulong gdb_get_module_boundary(const char *, bool); >>> ++#endif >>> ++ >>> + /* Find the compunit symtab associated with PC and SECTION. >>> + This will read in debug info as necessary. */ >>> + >>> +@@ -2932,11 +2936,21 @@ find_pc_sect_compunit_symtab (CORE_ADDR pc, struct obj_section *section) >>> + for (objfile *obj_file : current_program_space->objfiles ()) >>> + { >>> + #ifdef CRASH_MERGE >>> ++ size_t sep; >>> + std::string objfile_name = objfile_filename(obj_file); >>> ++ const char *mod_name; >>> + >>> + if (objfile_name.find(".ko") != std::string::npos) { >>> + if (obj_file->sf && obj_file->compunit_symtabs == nullptr) >>> + obj_file->sf->qf->expand_all_symtabs(obj_file); >>> ++ sep = objfile_name.find_last_of("/"); >>> ++ objfile_name = objfile_name.substr(sep + 1, objfile_name.size() - sep - 1); >>> ++ sep = objfile_name.find_first_of("."); >>> ++ objfile_name = objfile_name.substr(0, sep); >>> ++ mod_name = objfile_name.c_str(); >>> ++ if (pc < gdb_get_module_boundary(mod_name, true) || >>> ++ pc > gdb_get_module_boundary(mod_name, false)) >>> ++ continue; >>> + } >>> + #endif >>> + for (compunit_symtab *cust : obj_file->compunits ()) >>> diff --git a/gdb_interface.c b/gdb_interface.c >>> index b14319c..a58fd68 100644 >>> --- a/gdb_interface.c >>> +++ b/gdb_interface.c >>> @@ -947,6 +947,24 @@ gdb_lookup_module_symbol(ulong addr, ulong *offset) >>> } >>> } >>> >>> +ulong >>> +gdb_get_module_boundary(const char *module, bool is_start) >>> +{ >>> + char buf[256]; >>> + struct syment *sp; >>> + >>> + if (is_start) { >>> + snprintf(buf, sizeof(buf), "_MODULE_START_%s", module); >>> + } else { >>> + snprintf(buf, sizeof(buf), "_MODULE_END_%s", module); >>> + } >>> + sp = symbol_search(buf); >>> + if (sp) >>> + return sp->value; >>> + else >>> + return is_start ? 0 : (ulong)(-1); >>> +} >>> + >>> /* >>> * Used by gdb_interface() to catch gdb-related errors, if desired. >>> */

HAGIO KAZUHITO(萩尾　一仁)

Wednesday, 8 November Wed, 8 Nov

2:16 a.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

On 2023/11/08 12:01, HAGIO KAZUHITO(萩尾一仁) wrote:

...

Hmm, it looks like several sections are mapped outside of the current module range. What if they are not specified or they are mapped to e.g. an unused range..? or any idea? crash-dev> sym -m floppy | grep MODULE ffffffffc0092000 MODULE START: floppy ffffffffc00a2f29 MODULE END: floppy crash-dev> set debug 1 debug: 1 crash-dev> mod -s floppy ffffffffc009a000 .note.gnu.build-id ffffffffc0092000 .text ffffffffc00a4000 .init.text <<< higher than MODULE END ffffffffc009971d .text.unlikely ffffffffc00998ea .exit.text ffffffffc009a040 .rodata ffffffffc009a4d0 .rodata.str1.1 ffffffffc009ad98 .rodata.str1.8 ffffffffc009be14 .smp_locks ffffffffc009be98 .parainstructions ffffffffc009beb8 __param ffffffffc009bf18 __mcount_loc ffffffffc009d000 .data ffffffffc00a5000 .init.data <<< ffffffffc009db00 .gnu.linkonce.this_module ffffffffc009dd40 .bss ffffffffc00a6000 .symtab <<< ffffffffc00a80b8 .strtab <<< add-symbol-file /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug 0xffffffffc0092000 -s .note.gnu.build-id 0xffffffffc009a000 -s .init.text 0xffffffffc00a4000 -s .text.unlikely 0xffffffffc009971d -s .exit.text 0xffffffffc00998ea -s .rodata 0xffffffffc009a040 -s .rodata.str1.1 0xffffffffc009a4d0 -s .rodata.str1.8 0xffffffffc009ad98 -s .smp_locks 0xffffffffc009be14 -s .parainstructions 0xffffffffc009be98 -s __param 0xffffffffc009beb8 -s __mcount_loc 0xffffffffc009bf18 -s .data 0xffffffffc009d000 -s .init.data 0xffffffffc00a5000 -s .gnu.linkonce.this_module 0xffffffffc009db00 -s .bss 0xffffffffc009dd40 -s .symtab 0xffffffffc00a6000 -s .strtab 0xffffffffc00a80b8 add symbol table from file "/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug" at .text_addr = 0xffffffffc0092000 .note.gnu.build-id_addr = 0xffffffffc009a000 .init.text_addr = 0xffffffffc00a4000 .text.unlikely_addr = 0xffffffffc009971d .exit.text_addr = 0xffffffffc00998ea .rodata_addr = 0xffffffffc009a040 .rodata.str1.1_addr = 0xffffffffc009a4d0 .rodata.str1.8_addr = 0xffffffffc009ad98 .smp_locks_addr = 0xffffffffc009be14 .parainstructions_addr = 0xffffffffc009be98 __param_addr = 0xffffffffc009beb8 __mcount_loc_addr = 0xffffffffc009bf18 .data_addr = 0xffffffffc009d000 .init.data_addr = 0xffffffffc00a5000 .gnu.linkonce.this_module_addr = 0xffffffffc009db00 .bss_addr = 0xffffffffc009dd40 .symtab_addr = 0xffffffffc00a6000 .strtab_addr = 0xffffffffc00a80b8 warning: section .symtab not found in /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug warning: section .strtab not found in /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug MODULE NAME BASE SIZE OBJECT FILE ffffffffc009db00 floppy ffffffffc0092000 69417 /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug Thanks, Kazu > > Thanks, > Kazu > > > On 2023/11/07 17:20, Tao Liu wrote: >> Hi Kazu, >> >> On Tue, Nov 7, 2023 at 1:07 PM HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com> wrote: >>> >>> Hi Tao, >>> >>> thank you for the patches. >>> >>> On 2023/11/02 21:09, Tao Liu wrote: >>>> The address range may overlap for kernel modules global block vector, >>>> for example: >>>> >>>> module compunit_symtab block start block end >>>> floppy floppy.c ffffffffc0092000 ffffffffc00a4fa6 >>>> virtio_blk virtio_blk.c ffffffffc00a4000 ffffffffc00fd080 >>> >>> hmm, this condition looks strange in the first place, is it inevitable? >>> In other words, I wonder if there might be something wrong with how to >>> load module symbol files. >>> >> >> Honestly I'm not sure why there is address overlap of the 2 modules >> global block, it shouldn't happen from my view. But it can happen on >> another kernel(4.18.0-425.13.1.el8_7.x86_64), see the results blow: >> >> module compunit_symtab block start block end >> libata.ko.debug libata-core.c ffffffffc054f000 ffffffffc0591b18 >> mgag200.ko.debug mgag200_drv.c ffffffffc03f3000 ffffffffc0595b0e >> >> module address boundary: >> crash> sym -l >> ffffffffc054f000 MODULE START: libata >> ...snip... >> ffffffffc0590000 MODULE END: libata >> ffffffffc0591000 MODULE START: mgag200 >> ...snip... >> ffffffffc059a000 MODULE END: mgag200 >> >>> What is the kernel version? >> >> The kernel which I used to create the patch is 3.10.0-693.2.2.el7.x86_64. >> >>> Do you have any patch to debug the gdb internal information? >> >> Sure, simple debug patch as follows: >> diff --git gdb-10.2/gdb/symtab.c.orig gdb-10.2/gdb/symtab.c >> index 82605cc..3af3570 100644 >> --- gdb-10.2/gdb/symtab.c.orig >> +++ gdb-10.2/gdb/symtab.c >> @@ -2939,6 +2939,8 @@ find_pc_sect_compunit_symtab (CORE_ADDR pc, >> struct obj_section *section) >> bv = COMPUNIT_BLOCKVECTOR (cust); >> b = BLOCKVECTOR_BLOCK (bv, GLOBAL_BLOCK); >> >> + printf("%s %s %lx %lx\n", objfile_filename(obj_file), >> cust->name, BLOCK_START (b), BLOCK_END (b)); >> + >> if (BLOCK_START (b) <= pc >> && BLOCK_END (b) > pc >> && (distance == 0 >>> >>> I'd like to check vmcores on hand.. >> >> I can share the vmcore with you by a private email later. >> >> Thanks, >> Tao Liu >> >>> >>> Thanks, >>> Kazu >>> >>>> >>>> According to the gdb source code, the distance(block end - block start) of >>>> floppy(0x12fa6) is smaller than virtio_blk(0x59080), so address >>>> 0xffffffffc00a45b0, 0xffffffffc00a4370, 0xffffffffc00a4260 are translated >>>> by floppy module instead of virtio_blk module. However according >>>> to crash, the address range for the 2 modules are: >>>> >>>> crash> sym -l >>>> ffffffffc0092000 MODULE START: floppy >>>> ...snip... >>>> ffffffffc00a2f29 MODULE END: floppy >>>> ffffffffc00a4000 MODULE START: virtio_blk >>>> ...snip... >>>> ffffffffc00a86ec MODULE END: virtio_blk >>>> >>>> So the block vector address arrange and distance is not a reliable way to >>>> determine the symbol belonging of a specific kernel module. This patch will >>>> let gdb to query the module address range from crash, in order to locate the >>>> symbol belongings more percisely. >>>> >>>> Without the patch: >>>> crash> mod -S >>>> crash> struct blk_mq_ops 0xffffffffc00a7160 >>>> struct blk_mq_ops { >>>> queue_rq = 0xffffffffc00a45b0 <floppy_module_init+1151>, <-- symbol translated from module floppy >>>> map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, >>>> ...snip... >>>> complete = 0xffffffffc00a4370 <floppy_module_init+575>, >>>> init_request = 0xffffffffc00a4260 <floppy_module_init+303>, >>>> ...snip... >>>> } >>>> >>>> With the patch: >>>> crash> mod -S >>>> crash> struct blk_mq_ops 0xffffffffc00a7160 >>>> struct blk_mq_ops { >>>> queue_rq = 0xffffffffc00a45b0 <virtio_queue_rq>, <-- symbol translated from module virtio_blk >>>> map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, >>>> ...snip... >>>> complete = 0xffffffffc00a4370 <virtblk_request_done>, >>>> init_request = 0xffffffffc00a4260 <virtblk_init_request>, >>>> ...snip... >>>> } >>>> >>>> Signed-off-by: Tao Liu <ltao(a)redhat.com> >>>> --- >>>> defs.h | 2 ++ >>>> gdb-10.2.patch | 37 ++++++++++++++++++++++++++++++++++++- >>>> gdb_interface.c | 18 ++++++++++++++++++ >>>> 3 files changed, 56 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/defs.h b/defs.h >>>> index 788f63a..09e6738 100644 >>>> --- a/defs.h >>>> +++ b/defs.h >>>> @@ -5130,6 +5130,7 @@ struct syment *symbol_search(char *); >>>> int gdb_line_number_callback(ulong, ulong, ulong); >>>> int gdb_print_callback(ulong); >>>> char *gdb_lookup_module_symbol(ulong, ulong *); >>>> +ulong gdb_get_module_boundary(const char *, bool); >>>> extern "C" int same_file(char *, char *); >>>> #endif >>>> >>>> @@ -7687,6 +7688,7 @@ int gdb_readmem_callback(ulong, void *, int, int); >>>> int gdb_line_number_callback(ulong, ulong, ulong); >>>> int gdb_print_callback(ulong); >>>> char *gdb_lookup_module_symbol(ulong, ulong *); >>>> +ulong gdb_get_module_boundary(const char *, bool); >>>> void gdb_error_hook(void); >>>> void restore_gdb_sanity(void); >>>> int is_gdb_command(int, ulong); >>>> diff --git a/gdb-10.2.patch b/gdb-10.2.patch >>>> index 31135ca..a5c10d3 100644 >>>> --- a/gdb-10.2.patch >>>> +++ b/gdb-10.2.patch >>>> @@ -3203,4 +3203,39 @@ exit 0 >>>> +#endif >>>> for (compunit_symtab *cust : obj_file->compunits ()) >>>> { >>>> - const struct block *b; >>>> \ No newline at end of file >>>> + const struct block *b; >>>> +--- gdb-10.2/gdb/symtab.c.orig >>>> ++++ gdb-10.2/gdb/symtab.c >>>> +@@ -2895,6 +2895,10 @@ iterate_over_symbols_terminated >>>> + return callback (&block_sym); >>>> + } >>>> + >>>> ++#ifdef CRASH_MERGE >>>> ++extern "C" ulong gdb_get_module_boundary(const char *, bool); >>>> ++#endif >>>> ++ >>>> + /* Find the compunit symtab associated with PC and SECTION. >>>> + This will read in debug info as necessary. */ >>>> + >>>> +@@ -2932,11 +2936,21 @@ find_pc_sect_compunit_symtab (CORE_ADDR pc, struct obj_section *section) >>>> + for (objfile *obj_file : current_program_space->objfiles ()) >>>> + { >>>> + #ifdef CRASH_MERGE >>>> ++ size_t sep; >>>> + std::string objfile_name = objfile_filename(obj_file); >>>> ++ const char *mod_name; >>>> + >>>> + if (objfile_name.find(".ko") != std::string::npos) { >>>> + if (obj_file->sf && obj_file->compunit_symtabs == nullptr) >>>> + obj_file->sf->qf->expand_all_symtabs(obj_file); >>>> ++ sep = objfile_name.find_last_of("/"); >>>> ++ objfile_name = objfile_name.substr(sep + 1, objfile_name.size() - sep - 1); >>>> ++ sep = objfile_name.find_first_of("."); >>>> ++ objfile_name = objfile_name.substr(0, sep); >>>> ++ mod_name = objfile_name.c_str(); >>>> ++ if (pc < gdb_get_module_boundary(mod_name, true) || >>>> ++ pc > gdb_get_module_boundary(mod_name, false)) >>>> ++ continue; >>>> + } >>>> + #endif >>>> + for (compunit_symtab *cust : obj_file->compunits ()) >>>> diff --git a/gdb_interface.c b/gdb_interface.c >>>> index b14319c..a58fd68 100644 >>>> --- a/gdb_interface.c >>>> +++ b/gdb_interface.c >>>> @@ -947,6 +947,24 @@ gdb_lookup_module_symbol(ulong addr, ulong *offset) >>>> } >>>> } >>>> >>>> +ulong >>>> +gdb_get_module_boundary(const char *module, bool is_start) >>>> +{ >>>> + char buf[256]; >>>> + struct syment *sp; >>>> + >>>> + if (is_start) { >>>> + snprintf(buf, sizeof(buf), "_MODULE_START_%s", module); >>>> + } else { >>>> + snprintf(buf, sizeof(buf), "_MODULE_END_%s", module); >>>> + } >>>> + sp = symbol_search(buf); >>>> + if (sp) >>>> + return sp->value; >>>> + else >>>> + return is_start ? 0 : (ulong)(-1); >>>> +} >>>> + >>>> /* >>>> * Used by gdb_interface() to catch gdb-related errors, if desired. >>>> */ > -- > Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io > To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > Contribution Guidelines: https://github.com/crash-utility/crash/wiki

HAGIO KAZUHITO(萩尾　一仁)

Thursday, 9 November Thu, 9 Nov

2:37 a.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

On 2023/11/08 17:16, HAGIO KAZUHITO(萩尾一仁) wrote:

...

On 2023/11/08 12:01, HAGIO KAZUHITO(萩尾一仁) wrote: > Hi Tao, > > thank you for the information. > > I'm looking into it, I noticed that the unexpected symbol "floopy_module_init" > is in section .init.text. Crash side doesn't have the symbol info, probably > the address is already freed and reused by the virtio_blk module? > > crash> mod -S > ... > crash> sym -m floppy | grep MODULE > ffffffffc0092000 MODULE START: floppy > ffffffffc00a2f29 MODULE END: floppy > crash> sym -m virtio_blk | grep MODULE > ffffffffc00a4000 MODULE START: virtio_blk > ffffffffc00a86ec MODULE END: virtio_blk > crash> gdb info symbol floppy_module_init > init_module in section .init.text of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug > virtblk_freeze + 33 in section .text of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/virtio_blk.ko.debug > crash> gdb info address floppy_module_init > Symbol "floppy_module_init" is a function at address 0xffffffffc00a4131. > crash> sym floppy_module_init > symbol not found: floppy_module_init > possible alternatives: > (none found) > > So still not sure, but this might be an issue of handling the init > section/symbols of modules.. Hmm, it looks like several sections are mapped outside of the current module range. What if they are not specified or they are mapped to e.g. an unused range..? or any idea?

First, "mod -S -r" can reproduce this without the patch 1/2 or the commands. (If the root cause is fixed, I think no need to apply the patch 1/2.) crash-ups> mod -S -r crash-ups> p virtio_mq_ops virtio_mq_ops = $1 = { queue_rq = 0xffffffffc00a45b0 <floppy_module_init+1151>, map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, The following trial patch, which does not map the .init.text section specifically, fixes the issue. --- a/symbols.c +++ b/symbols.c @@ -13283,7 +13283,7 @@ add_symbol_file_kallsyms(struct load_module *lm, struct gnu_request *req) shift_string_right(req->buf, strlen(buf)); BCOPY(buf, req->buf, strlen(buf)); retval = TRUE; - } else { + } else if (!STREQ(section_name, ".init.text")) { sprintf(buf, " -s %s 0x%lx", section_name, section_vaddr); while ((len + strlen(buf)) >= buflen) { RESIZEBUF(req->buf, buflen, buflen * 2); crash-dev> mod -S -r crash-dev> p virtio_mq_ops virtio_mq_ops = $1 = { queue_rq = 0xffffffffc00a45b0 <virtio_queue_rq>, map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, { In this way, it looks like the .init.text sections of modules are mapped to 0x0, but many other sections like .modinfo are mapped here even without the trial patch. crash-dev> gdb info address floppy_module_init Symbol "floppy_module_init" is a function at address 0x131. crash-dev> gdb info symbol floppy_module_init irq_stack_union + 305 in section .data..percpu of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/vmlinux local_init + 305 in section .init.text of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/md/dm-mod.ko.debug ____versions + 49 in section __versions of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/md/dm-region-hash.ko.debug ____versions + 145 in section __versions of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/virtio/virtio.ko.debug __UNIQUE_ID_vermagic5 + 30 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/md/dm-log.ko.debug __UNIQUE_ID_srcversion7 + 1 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/md/dm-mirror.ko.debug __UNIQUE_ID_alias7 + 30 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/virtio/virtio_pci.ko.debug ____versions + 145 in section __versions of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/virtio/virtio_ring.ko.debug __UNIQUE_ID_alias16 + 36 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/ata/ata_generic.ko.debug ____versions + 17 in section __versions of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/fs/mbcache.ko.debug __UNIQUE_ID_vermagic5 + 45 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/net/virtio_net.ko.debug __UNIQUE_ID_vermagic5 + 44 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/cdrom/cdrom.ko.debug __UNIQUE_ID_vermagic5 + 14 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/scsi/sr_mod.ko.debug __module_depends + 1 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/net/ipv4/netfilter/ip_tables.ko.debug ____versions + 81 in section __versions of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/fs/binfmt_misc.ko.debug init_module in section .init.text of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug ... If a module already does not have its init memory range, it might be a bit better to not specify "-s .init.text <addr>" to add-symbol-file.. Thanks, Kazu > > crash-dev> sym -m floppy | grep MODULE > ffffffffc0092000 MODULE START: floppy > ffffffffc00a2f29 MODULE END: floppy > crash-dev> set debug 1 > debug: 1 > crash-dev> mod -s floppy > ffffffffc009a000 .note.gnu.build-id > ffffffffc0092000 .text > ffffffffc00a4000 .init.text <<< higher than MODULE END > ffffffffc009971d .text.unlikely > ffffffffc00998ea .exit.text > ffffffffc009a040 .rodata > ffffffffc009a4d0 .rodata.str1.1 > ffffffffc009ad98 .rodata.str1.8 > ffffffffc009be14 .smp_locks > ffffffffc009be98 .parainstructions > ffffffffc009beb8 __param > ffffffffc009bf18 __mcount_loc > ffffffffc009d000 .data > ffffffffc00a5000 .init.data <<< > ffffffffc009db00 .gnu.linkonce.this_module > ffffffffc009dd40 .bss > ffffffffc00a6000 .symtab <<< > ffffffffc00a80b8 .strtab <<< > add-symbol-file /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug 0xffffffffc0092000 -s .note.gnu.build-id 0xffffffffc009a000 -s .init.text 0xffffffffc00a4000 -s .text.unlikely 0xffffffffc009971d -s .exit.text 0xffffffffc00998ea -s .rodata 0xffffffffc009a040 -s .rodata.str1.1 0xffffffffc009a4d0 -s .rodata.str1.8 0xffffffffc009ad98 -s .smp_locks 0xffffffffc009be14 -s .parainstructions 0xffffffffc009be98 -s __param 0xffffffffc009beb8 -s __mcount_loc 0xffffffffc009bf18 -s .data 0xffffffffc009d000 -s .init.data 0xffffffffc00a5000 -s .gnu.linkonce.this_module 0xffffffffc009db00 -s .bss 0xffffffffc009dd40 -s .symtab 0xffffffffc00a6000 -s .strtab 0xffffffffc00a80b8 > add symbol table from file "/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug" at > .text_addr = 0xffffffffc0092000 > .note.gnu.build-id_addr = 0xffffffffc009a000 > .init.text_addr = 0xffffffffc00a4000 > .text.unlikely_addr = 0xffffffffc009971d > .exit.text_addr = 0xffffffffc00998ea > .rodata_addr = 0xffffffffc009a040 > .rodata.str1.1_addr = 0xffffffffc009a4d0 > .rodata.str1.8_addr = 0xffffffffc009ad98 > .smp_locks_addr = 0xffffffffc009be14 > .parainstructions_addr = 0xffffffffc009be98 > __param_addr = 0xffffffffc009beb8 > __mcount_loc_addr = 0xffffffffc009bf18 > .data_addr = 0xffffffffc009d000 > .init.data_addr = 0xffffffffc00a5000 > .gnu.linkonce.this_module_addr = 0xffffffffc009db00 > .bss_addr = 0xffffffffc009dd40 > .symtab_addr = 0xffffffffc00a6000 > .strtab_addr = 0xffffffffc00a80b8 > warning: section .symtab not found in /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug > warning: section .strtab not found in /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug > MODULE NAME BASE SIZE OBJECT FILE > ffffffffc009db00 floppy ffffffffc0092000 69417 /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug >

Tao Liu

2:44 a.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

Hi Kazu, Thanks a lot for your effort and comments. Sorry I was working on some other tasks yesterday and today. I will look into it later when my task is done. Thanks again for your help! Thanks, Tao Liu On Thu, Nov 9, 2023 at 4:37 PM HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com> wrote:

...

On 2023/11/08 17:16, HAGIO KAZUHITO(萩尾一仁) wrote: > On 2023/11/08 12:01, HAGIO KAZUHITO(萩尾一仁) wrote: >> Hi Tao, >> >> thank you for the information. >> >> I'm looking into it, I noticed that the unexpected symbol "floopy_module_init" >> is in section .init.text. Crash side doesn't have the symbol info, probably >> the address is already freed and reused by the virtio_blk module? >> >> crash> mod -S >> ... >> crash> sym -m floppy | grep MODULE >> ffffffffc0092000 MODULE START: floppy >> ffffffffc00a2f29 MODULE END: floppy >> crash> sym -m virtio_blk | grep MODULE >> ffffffffc00a4000 MODULE START: virtio_blk >> ffffffffc00a86ec MODULE END: virtio_blk >> crash> gdb info symbol floppy_module_init >> init_module in section .init.text of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug >> virtblk_freeze + 33 in section .text of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/virtio_blk.ko.debug >> crash> gdb info address floppy_module_init >> Symbol "floppy_module_init" is a function at address 0xffffffffc00a4131. >> crash> sym floppy_module_init >> symbol not found: floppy_module_init >> possible alternatives: >> (none found) >> >> So still not sure, but this might be an issue of handling the init >> section/symbols of modules.. > > Hmm, it looks like several sections are mapped outside of the current > module range. What if they are not specified or they are mapped to > e.g. an unused range..? or any idea? First, "mod -S -r" can reproduce this without the patch 1/2 or the commands. (If the root cause is fixed, I think no need to apply the patch 1/2.) crash-ups> mod -S -r crash-ups> p virtio_mq_ops virtio_mq_ops = $1 = { queue_rq = 0xffffffffc00a45b0 <floppy_module_init+1151>, map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, The following trial patch, which does not map the .init.text section specifically, fixes the issue. --- a/symbols.c +++ b/symbols.c @@ -13283,7 +13283,7 @@ add_symbol_file_kallsyms(struct load_module *lm, struct gnu_request *req) shift_string_right(req->buf, strlen(buf)); BCOPY(buf, req->buf, strlen(buf)); retval = TRUE; - } else { + } else if (!STREQ(section_name, ".init.text")) { sprintf(buf, " -s %s 0x%lx", section_name, section_vaddr); while ((len + strlen(buf)) >= buflen) { RESIZEBUF(req->buf, buflen, buflen * 2); crash-dev> mod -S -r crash-dev> p virtio_mq_ops virtio_mq_ops = $1 = { queue_rq = 0xffffffffc00a45b0 <virtio_queue_rq>, map_queue = 0xffffffff813015c0 <blk_mq_map_queue>, { In this way, it looks like the .init.text sections of modules are mapped to 0x0, but many other sections like .modinfo are mapped here even without the trial patch. crash-dev> gdb info address floppy_module_init Symbol "floppy_module_init" is a function at address 0x131. crash-dev> gdb info symbol floppy_module_init irq_stack_union + 305 in section .data..percpu of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/vmlinux local_init + 305 in section .init.text of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/md/dm-mod.ko.debug ____versions + 49 in section __versions of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/md/dm-region-hash.ko.debug ____versions + 145 in section __versions of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/virtio/virtio.ko.debug __UNIQUE_ID_vermagic5 + 30 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/md/dm-log.ko.debug __UNIQUE_ID_srcversion7 + 1 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/md/dm-mirror.ko.debug __UNIQUE_ID_alias7 + 30 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/virtio/virtio_pci.ko.debug ____versions + 145 in section __versions of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/virtio/virtio_ring.ko.debug __UNIQUE_ID_alias16 + 36 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/ata/ata_generic.ko.debug ____versions + 17 in section __versions of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/fs/mbcache.ko.debug __UNIQUE_ID_vermagic5 + 45 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/net/virtio_net.ko.debug __UNIQUE_ID_vermagic5 + 44 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/cdrom/cdrom.ko.debug __UNIQUE_ID_vermagic5 + 14 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/scsi/sr_mod.ko.debug __module_depends + 1 in section .modinfo of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/net/ipv4/netfilter/ip_tables.ko.debug ____versions + 81 in section __versions of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/fs/binfmt_misc.ko.debug init_module in section .init.text of /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug ... If a module already does not have its init memory range, it might be a bit better to not specify "-s .init.text <addr>" to add-symbol-file.. Thanks, Kazu > > crash-dev> sym -m floppy | grep MODULE > ffffffffc0092000 MODULE START: floppy > ffffffffc00a2f29 MODULE END: floppy > crash-dev> set debug 1 > debug: 1 > crash-dev> mod -s floppy > ffffffffc009a000 .note.gnu.build-id > ffffffffc0092000 .text > ffffffffc00a4000 .init.text <<< higher than MODULE END > ffffffffc009971d .text.unlikely > ffffffffc00998ea .exit.text > ffffffffc009a040 .rodata > ffffffffc009a4d0 .rodata.str1.1 > ffffffffc009ad98 .rodata.str1.8 > ffffffffc009be14 .smp_locks > ffffffffc009be98 .parainstructions > ffffffffc009beb8 __param > ffffffffc009bf18 __mcount_loc > ffffffffc009d000 .data > ffffffffc00a5000 .init.data <<< > ffffffffc009db00 .gnu.linkonce.this_module > ffffffffc009dd40 .bss > ffffffffc00a6000 .symtab <<< > ffffffffc00a80b8 .strtab <<< > add-symbol-file /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug 0xffffffffc0092000 -s .note.gnu.build-id 0xffffffffc009a000 -s .init.text 0xffffffffc00a4000 -s .text.unlikely 0xffffffffc009971d -s .exit.text 0xffffffffc00998ea -s .rodata 0xffffffffc009a040 -s .rodata.str1.1 0xffffffffc009a4d0 -s .rodata.str1.8 0xffffffffc009ad98 -s .smp_locks 0xffffffffc009be14 -s .parainstructions 0xffffffffc009be98 -s __param 0xffffffffc009beb8 -s __mcount_loc 0xffffffffc009bf18 -s .data 0xffffffffc009d000 -s .init.data 0xffffffffc00a5000 -s .gnu.linkonce.this_module 0xffffffffc009db00 -s .bss 0xffffffffc009dd40 -s .symtab 0xffffffffc00a6000 -s .strtab 0xffffffffc00a80b8 > add symbol table from file "/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug" at > .text_addr = 0xffffffffc0092000 > .note.gnu.build-id_addr = 0xffffffffc009a000 > .init.text_addr = 0xffffffffc00a4000 > .text.unlikely_addr = 0xffffffffc009971d > .exit.text_addr = 0xffffffffc00998ea > .rodata_addr = 0xffffffffc009a040 > .rodata.str1.1_addr = 0xffffffffc009a4d0 > .rodata.str1.8_addr = 0xffffffffc009ad98 > .smp_locks_addr = 0xffffffffc009be14 > .parainstructions_addr = 0xffffffffc009be98 > __param_addr = 0xffffffffc009beb8 > __mcount_loc_addr = 0xffffffffc009bf18 > .data_addr = 0xffffffffc009d000 > .init.data_addr = 0xffffffffc00a5000 > .gnu.linkonce.this_module_addr = 0xffffffffc009db00 > .bss_addr = 0xffffffffc009dd40 > .symtab_addr = 0xffffffffc00a6000 > .strtab_addr = 0xffffffffc00a80b8 > warning: section .symtab not found in /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug > warning: section .strtab not found in /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug > MODULE NAME BASE SIZE OBJECT FILE > ffffffffc009db00 floppy ffffffffc0092000 69417 /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug >

Tao Liu

9:15 p.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

Hi Kazu, On Thu, Nov 9, 2023 at 4:37 PM HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com> wrote:

...

Thanks a lot for finding the root cause. My patch is just a work-around, and I think your "trial" patch is better to be applied. I agree the ".init.text" should not be added by add-symbol-file, since this section will be freed and will be occupied by other modules after kernel init, which will cause symbols overlap. Could you please draft the "trial" patch to be the formal one? Thanks, Tao Liu

...

Thanks, Kazu > > crash-dev> sym -m floppy | grep MODULE > ffffffffc0092000 MODULE START: floppy > ffffffffc00a2f29 MODULE END: floppy > crash-dev> set debug 1 > debug: 1 > crash-dev> mod -s floppy > ffffffffc009a000 .note.gnu.build-id > ffffffffc0092000 .text > ffffffffc00a4000 .init.text <<< higher than MODULE END > ffffffffc009971d .text.unlikely > ffffffffc00998ea .exit.text > ffffffffc009a040 .rodata > ffffffffc009a4d0 .rodata.str1.1 > ffffffffc009ad98 .rodata.str1.8 > ffffffffc009be14 .smp_locks > ffffffffc009be98 .parainstructions > ffffffffc009beb8 __param > ffffffffc009bf18 __mcount_loc > ffffffffc009d000 .data > ffffffffc00a5000 .init.data <<< > ffffffffc009db00 .gnu.linkonce.this_module > ffffffffc009dd40 .bss > ffffffffc00a6000 .symtab <<< > ffffffffc00a80b8 .strtab <<< > add-symbol-file /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug 0xffffffffc0092000 -s .note.gnu.build-id 0xffffffffc009a000 -s .init.text 0xffffffffc00a4000 -s .text.unlikely 0xffffffffc009971d -s .exit.text 0xffffffffc00998ea -s .rodata 0xffffffffc009a040 -s .rodata.str1.1 0xffffffffc009a4d0 -s .rodata.str1.8 0xffffffffc009ad98 -s .smp_locks 0xffffffffc009be14 -s .parainstructions 0xffffffffc009be98 -s __param 0xffffffffc009beb8 -s __mcount_loc 0xffffffffc009bf18 -s .data 0xffffffffc009d000 -s .init.data 0xffffffffc00a5000 -s .gnu.linkonce.this_module 0xffffffffc009db00 -s .bss 0xffffffffc009dd40 -s .symtab 0xffffffffc00a6000 -s .strtab 0xffffffffc00a80b8 > add symbol table from file "/home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug" at > .text_addr = 0xffffffffc0092000 > .note.gnu.build-id_addr = 0xffffffffc009a000 > .init.text_addr = 0xffffffffc00a4000 > .text.unlikely_addr = 0xffffffffc009971d > .exit.text_addr = 0xffffffffc00998ea > .rodata_addr = 0xffffffffc009a040 > .rodata.str1.1_addr = 0xffffffffc009a4d0 > .rodata.str1.8_addr = 0xffffffffc009ad98 > .smp_locks_addr = 0xffffffffc009be14 > .parainstructions_addr = 0xffffffffc009be98 > __param_addr = 0xffffffffc009beb8 > __mcount_loc_addr = 0xffffffffc009bf18 > .data_addr = 0xffffffffc009d000 > .init.data_addr = 0xffffffffc00a5000 > .gnu.linkonce.this_module_addr = 0xffffffffc009db00 > .bss_addr = 0xffffffffc009dd40 > .symtab_addr = 0xffffffffc00a6000 > .strtab_addr = 0xffffffffc00a80b8 > warning: section .symtab not found in /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug > warning: section .strtab not found in /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug > MODULE NAME BASE SIZE OBJECT FILE > ffffffffc009db00 floppy ffffffffc0092000 69417 /home/vmcore/symbol_err/usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug >

HAGIO KAZUHITO(萩尾　一仁)

11:19 p.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

On 2023/11/10 12:15, Tao Liu wrote:

...

> If a module already does not have its init memory range, it might be > a bit better to not specify "-s .init.text <addr>" to add-symbol-file.. > Thanks a lot for finding the root cause. My patch is just a work-around, and I think your "trial" patch is better to be applied. I agree the ".init.text" should not be added by add-symbol-file, since this section will be freed and will be occupied by other modules after kernel init, which will cause symbols overlap. Could you please draft the "trial" patch to be the formal one?

The trial patch just doesn't add the .init.text unconditionally, but it should be required occasionally e.g. a panic when module initialization? As crash has a symbol table for a module init range: crash> help -s ... mod_base: ffffffffc0092000 module_struct: ffffffffc009db00 mod_name: floppy mod_size: 69417 mod_namelist: /home/vmcore/symbol_err/... mod_flags: 2a (MOD_LOAD_SYMS|MOD_KALLSYMS|MOD_NOPATCH) mod_symtable: 6a28170 mod_symend: 6a2a268 mod_init_symtable: 0 <<< mod_init_symend: 0 <<< So to finish a patch, we will need to look into a few more things: - It looks like the .init.{text,data} sections are always added on that kernel version, whether this is intended or not. For example, they might be wrongly added due to a name change of a struct member. - If it's an intended behavior, how we can exclude the .init sections only when it's unnecessary. and etc. Could you take a look? Thanks, Kazu

Tao Liu

Friday, 10 November Fri, 10 Nov

8:39 p.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

Hi Kazu, On Fri, Nov 10, 2023 at 1:19 PM HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com> wrote:

...

On 2023/11/10 12:15, Tao Liu wrote: >> If a module already does not have its init memory range, it might be >> a bit better to not specify "-s .init.text <addr>" to add-symbol-file.. >> > > Thanks a lot for finding the root cause. My patch is just a > work-around, and I think your "trial" patch is better to be applied. I > agree the ".init.text" should not be added by add-symbol-file, since > this section will be freed and will be occupied by other modules after > kernel init, which will cause symbols overlap. Could you please draft > the "trial" patch to be the formal one? The trial patch just doesn't add the .init.text unconditionally, but it should be required occasionally e.g. a panic when module initialization? As crash has a symbol table for a module init range: crash> help -s ... mod_base: ffffffffc0092000 module_struct: ffffffffc009db00 mod_name: floppy mod_size: 69417 mod_namelist: /home/vmcore/symbol_err/... mod_flags: 2a (MOD_LOAD_SYMS|MOD_KALLSYMS|MOD_NOPATCH) mod_symtable: 6a28170 mod_symend: 6a2a268 mod_init_symtable: 0 <<< mod_init_symend: 0 <<< So to finish a patch, we will need to look into a few more things: - It looks like the .init.{text,data} sections are always added on that kernel version, whether this is intended or not. For example, they might be wrongly added due to a name change of a struct member. - If it's an intended behavior, how we can exclude the .init sections only when it's unnecessary.

I agree the .init.{text,data} sections should be loaded if the module crashes during init. Here is a trial patch which worked for me: diff --git a/symbols.c b/symbols.c index 8e8b4c3..24b879d 100644 --- a/symbols.c +++ b/symbols.c @@ -13096,6 +13096,7 @@ add_symbol_file_kallsyms(struct load_module *lm, struct gnu_request *req) char buf[BUFSIZE]; char section_name[BUFSIZE/2]; ulong section_vaddr; + long mod_init_module_ptr; #if defined(GDB_5_3) || defined(GDB_6_0) || defined(GDB_6_1) return FALSE; @@ -13191,6 +13192,10 @@ add_symbol_file_kallsyms(struct load_module *lm, struct gnu_request *req) req->buf = GETBUF(buflen = 1024); retval = FALSE; + readmem(lm->module_struct + MODULE_OFFSET2(module_module_init, rx), KVADDR, + &mod_init_module_ptr, sizeof(void *), "module.module_init", FAULT_ON_ERROR); + fprintf(fp, ">>>>>>>> %lx\n", mod_init_module_ptr); + for (done = FALSE; !done; array_entry += SIZE(module_sect_attr)) { switch (st->flags & MODSECT_VMASK) @@ -13283,7 +13288,7 @@ add_symbol_file_kallsyms(struct load_module *lm, struct gnu_request *req) shift_string_right(req->buf, strlen(buf)); BCOPY(buf, req->buf, strlen(buf)); retval = TRUE; - } else { + } else if (mod_init_module_ptr || strncmp(section_name, ".init.", 6)) { sprintf(buf, " -s %s 0x%lx", section_name, section_vaddr); while ((len + strlen(buf)) >= buflen) { RESIZEBUF(req->buf, buflen, buflen * 2); According to kernel source kernel/module.c:do_init_module, after module been init successfully, module_init etc will be freed, and it will be reset to NULL: unset_module_init_ro_nx(mod); module_free(mod, mod->module_init); mod->module_init = NULL; <<---------- mod->init_size = 0; mod->init_ro_size = 0; mod->init_text_size = 0; So in crash, if the module_init member of struct module is NULL, we can be sure this module has been init successfully and the .init. section has been freed, so there is no need to load .init. section by add-symbol-file. However if we encounter the module_init with a value, the .init. section is not freed and should be loaded by add-symbol-file. Here is a kernel module which will fail during init for testing: hello/hello.c: static int __init hello_world_init(void) { int ret = 0; printk(KERN_INFO "Hello, World!\n"); *(int *)ret = 100; return 0; // Initialization successful } crash> mod -s hello /root/hello/hello.ko

...

>>>>>>> ffffffffc05e0000 <<----------module_init with value

MODULE NAME BASE SIZE OBJECT FILE ffffffffc05dd000 hello ffffffffc05db000 12431 /root/hello/hello.ko What do you think of the solution? Thanks, Tao Liu

...

and etc. Could you take a look? Thanks, Kazu

HAGIO KAZUHITO(萩尾　一仁)

Monday, 13 November Mon, 13 Nov

12:46 a.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

On 2023/11/11 11:39, Tao Liu wrote:

...

Hi Kazu, On Fri, Nov 10, 2023 at 1:19 PM HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com> wrote: > > On 2023/11/10 12:15, Tao Liu wrote: > >>> If a module already does not have its init memory range, it might be >>> a bit better to not specify "-s .init.text <addr>" to add-symbol-file.. >>> >> >> Thanks a lot for finding the root cause. My patch is just a >> work-around, and I think your "trial" patch is better to be applied. I >> agree the ".init.text" should not be added by add-symbol-file, since >> this section will be freed and will be occupied by other modules after >> kernel init, which will cause symbols overlap. Could you please draft >> the "trial" patch to be the formal one? > > The trial patch just doesn't add the .init.text unconditionally, but it > should be required occasionally e.g. a panic when module initialization? > As crash has a symbol table for a module init range: > > crash> help -s > ... > mod_base: ffffffffc0092000 > module_struct: ffffffffc009db00 > mod_name: floppy > mod_size: 69417 > mod_namelist: /home/vmcore/symbol_err/... > mod_flags: 2a (MOD_LOAD_SYMS|MOD_KALLSYMS|MOD_NOPATCH) > mod_symtable: 6a28170 > mod_symend: 6a2a268 > mod_init_symtable: 0 <<< > mod_init_symend: 0 <<< > > > So to finish a patch, we will need to look into a few more things: > > - It looks like the .init.{text,data} sections are always added on that > kernel version, whether this is intended or not. For example, they > might be wrongly added due to a name change of a struct member. > > - If it's an intended behavior, how we can exclude the .init sections > only when it's unnecessary. > I agree the .init.{text,data} sections should be loaded if the module crashes during init. Here is a trial patch which worked for me: diff --git a/symbols.c b/symbols.c index 8e8b4c3..24b879d 100644 --- a/symbols.c +++ b/symbols.c @@ -13096,6 +13096,7 @@ add_symbol_file_kallsyms(struct load_module *lm, struct gnu_request *req) char buf[BUFSIZE]; char section_name[BUFSIZE/2]; ulong section_vaddr; + long mod_init_module_ptr; #if defined(GDB_5_3) || defined(GDB_6_0) || defined(GDB_6_1) return FALSE; @@ -13191,6 +13192,10 @@ add_symbol_file_kallsyms(struct load_module *lm, struct gnu_request *req) req->buf = GETBUF(buflen = 1024); retval = FALSE; + readmem(lm->module_struct + MODULE_OFFSET2(module_module_init, rx), KVADDR, + &mod_init_module_ptr, sizeof(void *), "module.module_init", FAULT_ON_ERROR); + fprintf(fp, ">>>>>>>> %lx\n", mod_init_module_ptr); + for (done = FALSE; !done; array_entry += SIZE(module_sect_attr)) { switch (st->flags & MODSECT_VMASK) @@ -13283,7 +13288,7 @@ add_symbol_file_kallsyms(struct load_module *lm, struct gnu_request *req) shift_string_right(req->buf, strlen(buf)); BCOPY(buf, req->buf, strlen(buf)); retval = TRUE; - } else { + } else if (mod_init_module_ptr || strncmp(section_name, ".init.", 6)) { sprintf(buf, " -s %s 0x%lx", section_name, section_vaddr); while ((len + strlen(buf)) >= buflen) { RESIZEBUF(req->buf, buflen, buflen * 2); According to kernel source kernel/module.c:do_init_module, after module been init successfully, module_init etc will be freed, and it will be reset to NULL: unset_module_init_ro_nx(mod); module_free(mod, mod->module_init); mod->module_init = NULL; <<---------- mod->init_size = 0; mod->init_ro_size = 0; mod->init_text_size = 0; So in crash, if the module_init member of struct module is NULL, we can be sure this module has been init successfully and the .init. section has been freed, so there is no need to load .init. section by add-symbol-file. However if we encounter the module_init with a value, the .init. section is not freed and should be loaded by add-symbol-file. Here is a kernel module which will fail during init for testing: hello/hello.c: static int __init hello_world_init(void) { int ret = 0; printk(KERN_INFO "Hello, World!\n"); *(int *)ret = 100; return 0; // Initialization successful } crash> mod -s hello /root/hello/hello.ko >>>>>>>> ffffffffc05e0000 <<----------module_init with value MODULE NAME BASE SIZE OBJECT FILE ffffffffc05dd000 hello ffffffffc05db000 12431 /root/hello/hello.ko What do you think of the solution?

Thank you for considering this. I think a minimal change is better for now, so your solution looks fine to me. A few slight comments: - lm->mod_init_module_ptr should has the address if available, this can be used? - STRNEQ() is more preferable than direct strncmp() Thanks, Kazu

Tao Liu

7:09 a.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

Hi Kazu, On Mon, Nov 13, 2023 at 2:46 PM HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com> wrote:

...

On 2023/11/11 11:39, Tao Liu wrote: > Hi Kazu, > > On Fri, Nov 10, 2023 at 1:19 PM HAGIO KAZUHITO(萩尾　一仁) > <k-hagio-ab(a)nec.com> wrote: >> >> On 2023/11/10 12:15, Tao Liu wrote: >> >>>> If a module already does not have its init memory range, it might be >>>> a bit better to not specify "-s .init.text <addr>" to add-symbol-file.. >>>> >>> >>> Thanks a lot for finding the root cause. My patch is just a >>> work-around, and I think your "trial" patch is better to be applied. I >>> agree the ".init.text" should not be added by add-symbol-file, since >>> this section will be freed and will be occupied by other modules after >>> kernel init, which will cause symbols overlap. Could you please draft >>> the "trial" patch to be the formal one? >> >> The trial patch just doesn't add the .init.text unconditionally, but it >> should be required occasionally e.g. a panic when module initialization? >> As crash has a symbol table for a module init range: >> >> crash> help -s >> ... >> mod_base: ffffffffc0092000 >> module_struct: ffffffffc009db00 >> mod_name: floppy >> mod_size: 69417 >> mod_namelist: /home/vmcore/symbol_err/... >> mod_flags: 2a (MOD_LOAD_SYMS|MOD_KALLSYMS|MOD_NOPATCH) >> mod_symtable: 6a28170 >> mod_symend: 6a2a268 >> mod_init_symtable: 0 <<< >> mod_init_symend: 0 <<< >> >> >> So to finish a patch, we will need to look into a few more things: >> >> - It looks like the .init.{text,data} sections are always added on that >> kernel version, whether this is intended or not. For example, they >> might be wrongly added due to a name change of a struct member. >> >> - If it's an intended behavior, how we can exclude the .init sections >> only when it's unnecessary. >> > I agree the .init.{text,data} sections should be loaded if the module > crashes during init. > > Here is a trial patch which worked for me: > > diff --git a/symbols.c b/symbols.c > index 8e8b4c3..24b879d 100644 > --- a/symbols.c > +++ b/symbols.c > @@ -13096,6 +13096,7 @@ add_symbol_file_kallsyms(struct load_module > *lm, struct gnu_request *req) > char buf[BUFSIZE]; > char section_name[BUFSIZE/2]; > ulong section_vaddr; > + long mod_init_module_ptr; > > #if defined(GDB_5_3) || defined(GDB_6_0) || defined(GDB_6_1) > return FALSE; > @@ -13191,6 +13192,10 @@ add_symbol_file_kallsyms(struct load_module > *lm, struct gnu_request *req) > req->buf = GETBUF(buflen = 1024); > retval = FALSE; > > + readmem(lm->module_struct + MODULE_OFFSET2(module_module_init, > rx), KVADDR, > + &mod_init_module_ptr, sizeof(void *), "module.module_init", > FAULT_ON_ERROR); > + fprintf(fp, ">>>>>>>> %lx\n", mod_init_module_ptr); > + > for (done = FALSE; !done; array_entry += SIZE(module_sect_attr)) { > > switch (st->flags & MODSECT_VMASK) > @@ -13283,7 +13288,7 @@ add_symbol_file_kallsyms(struct load_module > *lm, struct gnu_request *req) > shift_string_right(req->buf, strlen(buf)); > BCOPY(buf, req->buf, strlen(buf)); > retval = TRUE; > - } else { > + } else if (mod_init_module_ptr || > strncmp(section_name, ".init.", 6)) { > sprintf(buf, " -s %s 0x%lx", section_name, > section_vaddr); > while ((len + strlen(buf)) >= buflen) { > RESIZEBUF(req->buf, buflen, buflen * 2); > > According to kernel source kernel/module.c:do_init_module, after > module been init successfully, module_init etc will be freed, and it > will be reset to NULL: > > unset_module_init_ro_nx(mod); > module_free(mod, mod->module_init); > mod->module_init = NULL; <<---------- > mod->init_size = 0; > mod->init_ro_size = 0; > mod->init_text_size = 0; > > So in crash, if the module_init member of struct module is NULL, we > can be sure this module has been init successfully and the .init. > section has been freed, so there is no need to load .init. section by > add-symbol-file. However if we encounter the module_init with a value, > the .init. section is not freed and should be loaded by > add-symbol-file. > > Here is a kernel module which will fail during init for testing: > > hello/hello.c: > static int __init hello_world_init(void) { > int ret = 0; > printk(KERN_INFO "Hello, World!\n"); > *(int *)ret = 100; > > return 0; // Initialization successful > } > > crash> mod -s hello /root/hello/hello.ko >>>>>>>>> ffffffffc05e0000 <<----------module_init with value > MODULE NAME BASE > SIZE OBJECT FILE > ffffffffc05dd000 hello ffffffffc05db000 > 12431 /root/hello/hello.ko > > What do you think of the solution? Thank you for considering this. I think a minimal change is better for now, so your solution looks fine to me.

Thanks for providing the feedback!

...

A few slight comments: - lm->mod_init_module_ptr should has the address if available, this can be used?

Sorry I didn't get your point. The value here is only used as a flag to indicate if the .init. section has been freed or not. Since the value is important, other place in crash also used it and read it there, symbol.c:store_module_symbols_v2() . Do you mean to share the value in different places so crash only read it once?

...

- STRNEQ() is more preferable than direct strncmp()

Agreed, will get it updated in v2. Thanks, Tao Liu

...

Thanks, Kazu

HAGIO KAZUHITO(萩尾　一仁)

Tuesday, 14 November Tue, 14 Nov

12:41 a.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

On 2023/11/13 22:09, Tao Liu wrote:

...

> A few slight comments: > > - lm->mod_init_module_ptr should has the address if available, this can > be used? Sorry I didn't get your point. The value here is only used as a flag to indicate if the .init. section has been freed or not. Since the value is important, other place in crash also used it and read it there, symbol.c:store_module_symbols_v2() . Do you mean to share the value in different places so crash only read it once?

Yes. For dumpfiles, it's always ok to use lm->mod_init_module_ptr. For live systems, if there is a change in module load status when "mod" command is run, reinit_modules() is executed. I think that a contradiction rarely happens. If it happens, crash will has internal contradictions (e.g. between "help -s" output and its load status) with the readmem() in add_symbol_file_kallsyms(). So I would prefer using lm->mod_init_module_ptr for consistency. Thanks, Kazu >> - STRNEQ() is more preferable than direct strncmp() >> > Agreed, will get it updated in v2. > > Thanks, > Tao Liu > >> Thanks, >> Kazu

Tao Liu

2:31 a.m.

New subject: [PATCH 2/2] symbols: fix the error belonging of the kernel modules symbols

Hi Kazu, On Tue, Nov 14, 2023 at 2:41 PM HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com> wrote:

...

On 2023/11/13 22:09, Tao Liu wrote: >> A few slight comments: >> >> - lm->mod_init_module_ptr should has the address if available, this can >> be used? > Sorry I didn't get your point. The value here is only used as a flag > to indicate if the .init. section has been freed or not. Since the > value is important, other place in crash also used it and read it > there, symbol.c:store_module_symbols_v2() . Do you mean to share the > value in different places so crash only read it once? Yes. For dumpfiles, it's always ok to use lm->mod_init_module_ptr. For live systems, if there is a change in module load status when "mod" command is run, reinit_modules() is executed. I think that a contradiction rarely happens. If it happens, crash will has internal contradictions (e.g. between "help -s" output and its load status) with the readmem() in add_symbol_file_kallsyms(). So I would prefer using lm->mod_init_module_ptr for consistency.

Agreed, thanks for the explanation. Will get it updated in v2. Thanks, Tao Liu

...

Thanks, Kazu >> - STRNEQ() is more preferable than direct strncmp() >> > Agreed, will get it updated in v2. > > Thanks, > Tao Liu > >> Thanks, >> Kazu

378

days inactive

390

days old

devel@lists.crash-utility.osci.io

Manage subscription

14 comments

2 participants

tags (0)

participants (2)

HAGIO KAZUHITO(萩尾　一仁)
Tao Liu

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

[PATCH 1/2] symbols: expand kernel modules symtable before symbols translation