Hi All When use bt -f to show stack data, I need a quick way to find out what are these stack data. For example, does any of these data are a inode pointer, or ... So here is always what i do. bt -f > stack kmem -S inode_cache > inode then use sort and comm utility to find value that appear in both files. Is there a better way to do this? I wish we can have a bt -f slab1 slab2... and try to match the stack data with content from these slab cache automatically. Thanks!

On Sat, 2008-03-01 at 17:15 -0800, James Washer wrote: > I think you'll have better luck consulting the source code, and > disassembly to figure out what is on the stack. checking asm code and finding out what is on the stack is sure the most accurate way. but a hint could speed up the process. > For the example you've cited, if a given function uses an inode pointer, > it shouldn't take but a minute or so to determine where in the stack > frame this inode is located. (unless of course it turns out to be in a > register only, in which case you have to look for it to be spilled in a > subsequent frame) quite some time you can see from the asm code that a important parameter is stored in register and you have to go several level deeper before finding it. so a hint which quickly locate a potential 'hot' data and later revalidate with code reading is still very appealing.

> - jim > > On Sat, 2008-03-01 at 17:38 -0500, Ming Zhang wrote: >> Hi All >> >> When use bt -f to show stack data, I need a quick way to find out what >> are these stack data. For example, does any of these data are a inode >> pointer, or ... So here is always what i do. >> >> bt -f > stack >> kmem -S inode_cache > inode >> >> then use sort and comm utility to find value that appear in both files. >> >> Is there a better way to do this? >> >> I wish we can have a bt -f slab1 slab2... >> >> and try to match the stack data with content from these slab cache >> automatically. >> >> Thanks! >> >>

On Mon, 2008-03-03 at 09:19 -0500, Dave Anderson wrote: > I'm not sure whether cluttering the bt -f command output would > be all that worthwhile, but I'm thinking that this request may have > some merit with respect to the "rd" command. Currently the > "rd -s" option recognizes and translates kernel symbols that it > finds in the raw memory output. Maybe something like a "-S" > option could do that -- plus also recognize kernel virtual memory > addresses that come from the slab cache, and alternatively display > the slab cache namestring in some recognizable manner, i.e. bracketed, > or something like to that effect. > > Then since "bt -f" displays the block of memory associated with > each stack frame, you could easily transfer the stack address > to a "rd -S [stack-address] [count]" command. yes. it will be great if we can have a rd -S!

On Mon, 2008-03-03 at 11:19 -0500, Dave Anderson wrote: > Ming Zhang wrote: >> On Mon, 2008-03-03 at 09:19 -0500, Dave Anderson wrote: >>> I'm not sure whether cluttering the bt -f command output would >>> be all that worthwhile, but I'm thinking that this request may have >>> some merit with respect to the "rd" command. Currently the >>> "rd -s" option recognizes and translates kernel symbols that it >>> finds in the raw memory output. Maybe something like a "-S" >>> option could do that -- plus also recognize kernel virtual memory >>> addresses that come from the slab cache, and alternatively display >>> the slab cache namestring in some recognizable manner, i.e. bracketed, >>> or something like to that effect. >>> >>> Then since "bt -f" displays the block of memory associated with >>> each stack frame, you could easily transfer the stack address >>> to a "rd -S [stack-address] [count]" command. >> yes. it will be great if we can have a rd -S! >> > Apply the attached patch and see what you think. The "rd -S" > option supplements "-s" by also recognizing memory from slab > caches -- and alternatively displaying the slab cache name > in brackets. > this is what it looks like 10078b2ba30: vprintk+498 .LC391+195657 10078b2ba40: 0000000000000471 [size-32] 10078b2ba50: 000000000000016e 0000000000000001 10078b2ba60: 0000000000000005 [kmem_cache] 10078b2ba70: 000000000000003c [size-1024] 10078b2ba80: [kmem_cache] 0000000000000000 10078b2ba90: 0000000000000001 00000000801108a1 [raw value: cache name] will be better

strangely, not all the value is correct. i am looking at it.

my English sucks... this is what i got. my side size-512 or size-1024 also works. just stuff like dentry cache or inode cache does not work. crash> rd 10078b2bca0 2 10078b2bca0: 000001007877c8d4 00000100710fd5c8 ..wx.......q.... crash> rd -S 10078b2bca0 2 10078b2bca0: [dentry_cache] 00000100710fd5c8 crash> kmem -s 10078b2bca0 kmem: address is not allocated in slab subsystem: 10078b2bca0

On Mon, 2008-03-03 at 14:04 -0500, Dave Anderson wrote: > Ming Zhang wrote: > >> my English sucks... >> >> this is what i got. my side size-512 or size-1024 also works. just stuff >> like dentry cache or inode cache does not work. >> >> >> crash> rd 10078b2bca0 2 >> 10078b2bca0: 000001007877c8d4 00000100710fd5c8 ..wx.......q.... >> crash> rd -S 10078b2bca0 2 >> 10078b2bca0: [dentry_cache] 00000100710fd5c8 >> crash> kmem -s 10078b2bca0 >> kmem: address is not allocated in slab subsystem: 10078b2bca0 > No, your English is fine -- it's your command-entering that sucks! > > Above, when you entered "kmem -s 10078b2bca0", you're incorrectly entering > the *address* where the dentry_cache reference (1007877c8d4) is located. > And so it's telling you that 10078b2bca0 is not allocated in the slab > subsystem, which it isn't... > > But if you entered "kmem -s 1007877c8d4", you'd see the dentry_cache > information. yes. i noticed that. need some tea when handling 3 bugs at the same time... so here is my understanding. it actually shows the slab object that contain that address. it might not be the address of that object. so here is what i need to do crash> rd -x 10078b2bca0 10078b2bca0: 000001007877c8d4 crash> rd -S 10078b2bca0 10078b2bca0: [dentry_cache] crash> kmem -s 000001007877c8d4 CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE 10037ffc080 dentry_cache 240 9429 10560 660 4k SLAB MEMORY TOTAL ALLOCATED FREE 1007877c040 1007877c088 16 9 7 FREE / [ALLOCATED] [1007877c8d4] this only tell me that it belongs to one dentry object, but no idea which one then i have to use kmem -S dentry_cache, find out this piece SLAB MEMORY TOTAL ALLOCATED FREE 1007877c040 1007877c088 16 9 7 FREE / [ALLOCATED] [1007877c088] [1007877c178] 1007877c268 (cpu 1 cache) 1007877c358 (cpu 1 cache) [1007877c448] 1007877c538 (cpu 1 cache) [1007877c628] [1007877c718] [1007877c808] ~~~~~~~~~~~~~~~~~~~~ then find out it is here. [1007877c8f8] 1007877c9e8 (cpu 1 cache) 1007877cad8 (cpu 1 cache) [1007877cbc8] [1007877ccb8] 1007877cda8 (cpu 1 cache) 1007877ce98 (cpu 1 cache) then i know the object contain this address is 1007877c808. then crash> dentry.d_iname struct dentry { [0xcc] unsigned char d_iname[36]; } crash> eval c808+cc hexadecimal: c8d4 decimal: 51412 octal: 144324 show me that variable in stack actually is d_iname. then can we have the output format as 10078b2bca0: [000001007877c808+cc: dentry_cache] so we know the object address, which slab it is in, and the offset, (thus can derive the raw value), all in one shot?

On Mon, 2008-03-03 at 14:56 -0500, Dave Anderson wrote: > Ming Zhang wrote: >> On Mon, 2008-03-03 at 14:04 -0500, Dave Anderson wrote: >>> Ming Zhang wrote: >>> >>>> my English sucks... >>>> >>>> this is what i got. my side size-512 or size-1024 also works. just stuff >>>> like dentry cache or inode cache does not work. >>>> >>>> >>>> crash> rd 10078b2bca0 2 >>>> 10078b2bca0: 000001007877c8d4 00000100710fd5c8 ..wx.......q.... >>>> crash> rd -S 10078b2bca0 2 >>>> 10078b2bca0: [dentry_cache] 00000100710fd5c8 >>>> crash> kmem -s 10078b2bca0 >>>> kmem: address is not allocated in slab subsystem: 10078b2bca0 >>> No, your English is fine -- it's your command-entering that sucks! >>> >>> Above, when you entered "kmem -s 10078b2bca0", you're incorrectly entering >>> the *address* where the dentry_cache reference (1007877c8d4) is located. >>> And so it's telling you that 10078b2bca0 is not allocated in the slab >>> subsystem, which it isn't... >>> >>> But if you entered "kmem -s 1007877c8d4", you'd see the dentry_cache >>> information. >> yes. i noticed that. need some tea when handling 3 bugs at the same >> time... >> >> so here is my understanding. it actually shows the slab object that >> contain that address. it might not be the address of that object. so >> here is what i need to do >> >> crash> rd -x 10078b2bca0 >> 10078b2bca0: 000001007877c8d4 >> crash> rd -S 10078b2bca0 >> 10078b2bca0: [dentry_cache] >> >> crash> kmem -s 000001007877c8d4 >> CACHE NAME OBJSIZE ALLOCATED TOTAL >> SLABS SSIZE >> 10037ffc080 dentry_cache 240 9429 10560 >> 660 4k >> SLAB MEMORY TOTAL ALLOCATED FREE >> 1007877c040 1007877c088 16 9 7 >> FREE / [ALLOCATED] >> [1007877c8d4] >> >> this only tell me that it belongs to one dentry object, but no idea >> which one >> >> then i have to use kmem -S dentry_cache, find out this piece >> >> SLAB MEMORY TOTAL ALLOCATED FREE >> 1007877c040 1007877c088 16 9 7 >> FREE / [ALLOCATED] >> [1007877c088] >> [1007877c178] >> 1007877c268 (cpu 1 cache) >> 1007877c358 (cpu 1 cache) >> [1007877c448] >> 1007877c538 (cpu 1 cache) >> [1007877c628] >> [1007877c718] >> [1007877c808] >> ~~~~~~~~~~~~~~~~~~~~ then find out it is here. >> [1007877c8f8] >> 1007877c9e8 (cpu 1 cache) >> 1007877cad8 (cpu 1 cache) >> [1007877cbc8] >> [1007877ccb8] >> 1007877cda8 (cpu 1 cache) >> 1007877ce98 (cpu 1 cache) >> >> then i know the object contain this address is 1007877c808. >> then >> >> crash> dentry.d_iname >> struct dentry { >> [0xcc] unsigned char d_iname[36]; >> } >> crash> eval c808+cc >> hexadecimal: c8d4 >> decimal: 51412 >> octal: 144324 >> >> show me that variable in stack actually is d_iname. >> >> then can we have the output format as >> >> 10078b2bca0: [000001007877c808+cc: dentry_cache] >> >> so we know the object address, which slab it is in, and the offset, >> (thus can derive the raw value), all in one shot? >> > You've done a fine bit of debugging your issue... > > But I think that's a bit of overkill for each address reference. > > To do it right it would have to walk slab chains to gather all of the > information needed by the "kmem -S" output, which can be extremely > time-consuming, and potentially error-prone if the slab chain is > corrupt or being modified while running on a live system. full agree. realized now. then here is my question, how "kmem -s <addr>" find out which slab this address belong to? by only looking at the page?

then here is my revised suggestion. can we split this into 2 steps? (1) rd -S show [raw address: cache name] (this is a great to have since do rd 2 times, one with -S and another without -S is tedious.)

(2) and kmem -s <address> show which slab it is in and optionally (with a new option like -O) show the object?

(this is nice to have). thanks.

On Mon, 2008-03-03 at 15:35 -0500, Dave Anderson wrote: <snip> >>>> so we know the object address, which slab it is in, and the offset, >>>> (thus can derive the raw value), all in one shot? >>>> >>> You've done a fine bit of debugging your issue... >>> >>> But I think that's a bit of overkill for each address reference. >>> >>> To do it right it would have to walk slab chains to gather all of the >>> information needed by the "kmem -S" output, which can be extremely >>> time-consuming, and potentially error-prone if the slab chain is >>> corrupt or being modified while running on a live system. >> full agree. realized now. >> >> then here is my question, how "kmem -s <addr>" find out which slab this >> address belong to? by only looking at the page? > The kmem_cache and containing slab addresses are stashed in unused > fields of the page structure of the object's virtual address. ic. good to know this. >> then here is my revised suggestion. can we split this into 2 steps? >> >> (1) rd -S show [raw address: cache name] >> >> (this is a great to have since do rd 2 times, one with -S and another >> without -S is tedious.) > But in your case, for example, I'm presuming you've done a "bt -f", > and simply would like a symbols/slab translation of a chunk of memory > making up one of the stack frames, so you do a subsequent "rd -S" of > that memory area. Doesn't seem that tedious to me... have to agree. >> (2) and kmem -s <address> >> >> show which slab it is in and optionally (with a new option like -O) show >> the object? > Not following -- "kmem -s <address>" does show which slab it's in. i emphasize on "and optionally...", ;) > And if you're asking whether the object can be dumped as the structure > type it is (?), well, how would it be possible from the crash utility's > viewpoint to even know what type of data structure it is? no. what i mean is kmem -s <address> does not give me the object address crash> kmem -s 000001007877c8d4 CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE 10037ffc080 dentry_cache 240 9429 10560 660 4k SLAB MEMORY TOTAL ALLOCATED FREE 1007877c040 1007877c088 16 9 7 FREE / [ALLOCATED] [1007877c8d4] i still have to use kmem -S to find out 1007877c808 is the object that contain this address. nice if crash can do this for me.

> If you want to look at all of the objects in a slab represented > as data structures, you're going to have supply the knowledge of > what data structure they are. It's simple enough, just do a "kmem -S" > into a file, delete everything except the object addresses that you're > interested in, insert "struct whatever" in front of each address, save this is exactly what i did when i have to do work like this by using gawk, tr, and grep. > the file -- and run it as crash input file. > how to do this? i know crash -i can run a file at beginning. but how to run command in a file at any moment?

> OK I understand. Yeah, it's always worked that way -- I don't recall > why other than the fact that by the time the address is displayed, the > function that does the display no longer has a handle on the beginning > address of the object, only the "requested" address, the slab it came > from, whether it's free/allocated, and whether it's sitting on a per-cpu > cache. I'll have to revisit that sometime... thanks for putting that on your todo list. so will you check in the patch soon?

>>> If you want to look at all of the objects in a slab represented >>> as data structures, you're going to have supply the knowledge of >>> what data structure they are. It's simple enough, just do a "kmem -S" >>> into a file, delete everything except the object addresses that you're >>> interested in, insert "struct whatever" in front of each address, save >> this is exactly what i did when i have to do work like this by using >> gawk, tr, and grep. >> >>> the file -- and run it as crash input file. >>> >> how to do this? i know crash -i can run a file at beginning. but how to >> run command in a file at any moment? >> > Enter "help input" -- where it talks about "<": thanks for the hint. you save me quite a lot key strokes! thanks again for all the help!