Re: [Crash-utility] Module debugging using crash

Dheeraj Sangamkar wrote: > > Hi, > Is there a way to print type information about types defined in a loadable > kernel module? Is there a way to add to the type information that crash gets > from the vmlinux file? > > This would ease debugging crashed module code be helping me print structures > defined in modules. > > Dheeraj Well, that's what the "mod" command does, i.e., load the debuginfo data of kernel modules into the embedded gdb module of the crash session. Take the "ext3" module for example. It declares this data structure in the file "fs/ext3/super.c": static struct super_operations ext3_sops = { .alloc_inode = ext3_alloc_inode, .destroy_inode = ext3_destroy_inode, .read_inode = ext3_read_inode, .write_inode = ext3_write_inode, .dirty_inode = ext3_dirty_inode, .delete_inode = ext3_delete_inode, .put_super = ext3_put_super, .write_super = ext3_write_super, .sync_fs = ext3_sync_fs, .write_super_lockfs = ext3_write_super_lockfs, .unlockfs = ext3_unlockfs, .statfs = ext3_statfs, .remount_fs = ext3_remount, .clear_inode = ext3_clear_inode, .show_options = ext3_show_options, #ifdef CONFIG_QUOTA .quota_read = ext3_quota_read, .quota_write = ext3_quota_write, #endif }; or for example, this data structure type is defined in "fs/ext3/xattr.h": struct ext3_xattr_header { __le32 h_magic; /* magic number for identification */ __le32 h_refcount; /* reference count */ __le32 h_blocks; /* number of disk blocks used */ __le32 h_hash; /* hash value of all attributes */ __u32 h_reserved[4]; /* zero right now */ }; Without the ext3 module's debuginfo data, crash (and gdb) doesn't know what they are, so if I try to print the "ext3_sops" symbolically, this happens: crash> p ext3_sops p: gdb request failed: p ext3_sops crash> whatis ext3_sops whatis: gdb request failed: whatis ext3_sops crash> And if I want to see what an ext3_xattr_header structure is defined as: crash> struct ext3_xattr_header struct: invalid data structure reference: ext3_xattr_header crash> But if the module's debuginfo data is loaded: crash> mod -s ext3 MODULE NAME SIZE OBJECT FILE e08c4f80 ext3 123337 /lib/modules/2.6.18-53.el5/kernel/fs/ext3/ext3.ko crash> (or use "mod -S" to load the debuginfo data of all modules), then the same commands that failed above work like so: crash> p ext3_sops ext3_sops = $4 = { alloc_inode = 0xe08b1c54 <ext3_alloc_inode>, destroy_inode = 0xe08b2e4a <ext3_destroy_inode>, read_inode = 0xe08ae0c9 <ext3_read_inode>, dirty_inode = 0xe08ae063 <ext3_dirty_inode>, write_inode = 0xe08ad696 <ext3_write_inode>, put_inode = 0, drop_inode = 0, delete_inode = 0xe08ad7af <ext3_delete_inode>, put_super = 0xe08b2f6d <ext3_put_super>, write_super = 0xe08b1c37 <ext3_write_super>, sync_fs = 0xe08b291d <ext3_sync_fs>, write_super_lockfs = 0xe08b295f <ext3_write_super_lockfs>, unlockfs = 0xe08b29ae <ext3_unlockfs>, statfs = 0xe08b2c55 <ext3_statfs>, remount_fs = 0xe08b2a21 <ext3_remount>, clear_inode = 0xe08b2e5a <ext3_clear_inode>, umount_begin = 0, show_options = 0xe08b2835 <ext3_show_options>, show_stats = 0, quota_read = 0xe08b449b <ext3_quota_read>, quota_write = 0xe08b45f5 <ext3_quota_write> } crash> crash> ext3_xattr_header struct ext3_xattr_header { __le32 h_magic; __le32 h_refcount; __le32 h_blocks; __le32 h_hash; __u32 h_reserved[4]; } SIZE: 32 crash> The example above presumes the Red Hat split debuginfo module format, where the "mod" command shows that the ext3 module exists in: /lib/modules/2.6.18-53.el5/kernel/fs/ext3/ext3.ko, but that ext3.ko module contains a link to its associated debuginfo part, which is actually located in: /usr/lib/debug/lib/modules/2.6.18-53.el5/kernel/fs/ext3/ext3.ko.debug If your module.ko file is not split into two, and has been built with -g, then you can load its debuginfo data crash like so: crash> mod -s yourmodule yourmodule.ko Dave