-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dave Anderson wrote: | Pete/Piet Delaney wrote: |> Got by 1st panic crash, crash seems to be working but |> I don't feel comfortable as with gdb. I can't see |> the parameters on the stack as variables as with |> kgdb nor can I see the local variables on the stack. |> Seems I should be able to get crash working with |> gdb so I can see the details that gdb can see. |> |> I compiled the kernel -O0 and without the framepointer |> as it seem crash prefers (rather odd). But want to |> see the locals and formals on the stack and walk up and |> down the stack and have the context know to crash just |> as with gdb. |> |> I'll re-read your crash paper. |> |> Any suggestions? | | Nope, other than using "bt -f" to dump each frame's data, | and figuring it out from there... When I was using gdb on SunOS 4.1.4 I was able to set $sp and $fp to switch processes. When I took the SP and FP and used a gdb set $sp and %fp it didn't seen to allow me to do this: - ----------------------------------------------------------- crash> gdb set $sp = 0xd76bbbb4 No registers. crash> gdb set $fp = 0xd76bbc24 No registers. - ----------------------------------------------------------- Why is is saying 'No registers'? Also why don't you set up gdb with the stack info for the current task and either allow the user to do a 'gdb bt' or do it with a crash 'bt' option? Will gdb work directly with the kdump core file? If not when did it break and why? Docs with 2.6.16 suggest to use gdb of the core file. I recall the crash backtrack proving the gdb info back a few years ago when I worked on LKCD and crash with you. Is my memory wrong and this visibility of the formals and local variables has never never been possible? It's hard to recall for sure with my using kgdb on the kernel now for so long. - -piet | | Dave | |

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dave Anderson wrote: | Pete/Piet Delaney wrote: | > -----BEGIN PGP SIGNED MESSAGE----- | > Hash: SHA1 | > | > Dave Anderson wrote: | > | Pete/Piet Delaney wrote: | > |> Got by 1st panic crash, crash seems to be working but | > |> I don't feel comfortable as with gdb. I can't see | > |> the parameters on the stack as variables as with | > |> kgdb nor can I see the local variables on the stack. | > |> Seems I should be able to get crash working with | > |> gdb so I can see the details that gdb can see. | > |> | > |> I compiled the kernel -O0 and without the framepointer | > |> as it seem crash prefers (rather odd). But want to | > |> see the locals and formals on the stack and walk up and | > |> down the stack and have the context know to crash just | > |> as with gdb. | > |> | > |> I'll re-read your crash paper. | > |> | > |> Any suggestions? | > | | > | Nope, other than using "bt -f" to dump each frame's data, | > | and figuring it out from there... | > | > When I was using gdb on SunOS 4.1.4 I was able to set $sp and | > $fp to switch processes. When I took the SP and FP and used | > a gdb set $sp and %fp it didn't seen to allow me to do this: | > - ----------------------------------------------------------- | > crash> gdb set $sp = 0xd76bbbb4 | > No registers. | > | > crash> gdb set $fp = 0xd76bbc24 | > No registers. | > - ----------------------------------------------------------- | > Why is is saying 'No registers'? | | I'm not positive, but the gdb sources seem to display that message | whenever (!target_has_registers), and that's #define'd like so: | | /* Does the target have registers? (Exec files don't.) */ | | #define target_has_registers \ | (current_target.to_has_registers) | | And since it's being invoked as "gdb vmlinux", it only knows | about the exec file, and has no clue about registers. Perhaps gdb changed since SunOS 4.1.4 in this area. We could consider patching gdb to think it has registers just to allow crash and/or the user from setting $sp and $fp to facilitate better back trace information. As I recall you make environment isn't building a virgin gdb. | > | > Also why don't you set up gdb with the stack info for the current | > task and either allow the user to do a 'gdb bt' or do it with a | > crash 'bt' option? | > | | I don't even come close to understanding gdb internals well enough | to know what "set up gdb with the stack info" entails. Send me | a patch when you've got it working -- for all arches, for all tasks, | and without requiring any vmcore file argument... ;-) | | > Will gdb work directly with the kdump core file? If not when did | > it break and why? Docs with 2.6.16 suggest to use gdb of the core | > file. | | I don't know if it *has* broke... | | As I said this morning, I don't know if the latest and greatest 32-bit | gdb can work with 64-bit ELF kdump vmcores, which kdump uses by default | for both 32-bit and 64-bit architectures. The kdump.txt file says this: | | * By default, the ELF headers are stored in ELF64 format to support | systems with more than 4GB memory. The --elf32-core-headers option can | be used to force the generation of ELF32 headers. This is necessary | because GDB currently cannot open vmcore files with ELF64 headers on | 32-bit systems. ELF32 headers can be used on non-PAE systems (that is, | less than 4GB of memory). | | But at least with a 32-bit ELF vmcore, gdb should be able to work. ok, I'll trying pulling out a DIMM and running on 2GB. | | > | > I recall the crash backtrack proving the gdb info back a few years | > ago when I worked on LKCD and crash with you. Is my memory wrong | > and this visibility of the formals and local variables has never | > never been possible? It's hard to recall for sure with my using | > kgdb on the kernel now for so long. | > | | Your memory's wrong, especially w/respect to local variables -- are | you kidding me? No, I wasn't joking, I thought back traces with crash had the same format as with gdb. Last I worked on IA64 and gdb I found messing with it's accessing of registers rather convolved. I was modifying Andy Kleen's gdb interface to SGI's kdb debugger, it's called 'SKDB'. At the time SGI had kdb working on IA64 kernels and I tried modifying if from ia32 to IA64. The register storage in gdb was a bit odd. Seems to be worthy of consideration to fool gdb into thing it has registers just so crash can get gdb to provide backtrace info. What happens of you simply change the #define to something like: #define target_has_registers 1 I take it that crash in doing the stack unwind it's self now instead of having gdb do it. Have you thought about having gdb do it and what's needed? On Friday you mentioned an embedded gdb module to hijack read attempts and back them to crash. I tried a Google for more info but didn't find anything. Could you tell me more about that? - -piet | | Dave | | -- | Crash-utility mailing list | Crash-utility(a)redhat.com | https://www.redhat.com/mailman/listinfo/crash-utility |

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dave Anderson wrote: | Pete/Piet Delaney wrote: |> | > When I was using gdb on SunOS 4.1.4 I was able to set $sp and |> | > $fp to switch processes. When I took the SP and FP and used |> | > a gdb set $sp and %fp it didn't seen to allow me to do this: |> | > - ----------------------------------------------------------- |> | > crash> gdb set $sp = 0xd76bbbb4 |> | > No registers. |> | > |> | > crash> gdb set $fp = 0xd76bbc24 |> | > No registers. |> | > - ----------------------------------------------------------- |> | > Why is is saying 'No registers'? |> | |> | I'm not positive, but the gdb sources seem to display that message |> | whenever (!target_has_registers), and that's #define'd like so: |> | |> | /* Does the target have registers? (Exec files don't.) */ |> | |> | #define target_has_registers \ |> | (current_target.to_has_registers) |> | |> | And since it's being invoked as "gdb vmlinux", it only knows |> | about the exec file, and has no clue about registers. |> |> Perhaps gdb changed since SunOS 4.1.4 in this area. We could |> consider patching gdb to think it has registers just to allow |> crash and/or the user from setting $sp and $fp to facilitate |> better back trace information. As I recall you make environment |> isn't building a virgin gdb. | | That's exactly what I tried many years ago -- just forcibly setting | the stack and frame pointer values. But it still required more | "user environment" information to work correctly. I had some limited | success on the alpha platform, but never on x86. | | And that's only the beginning. To be of any use, it would have to | be able to deal with in-kernel exception frames, which from a user | debugger standpoint, it has no idea what they are. And it would need | to be able to deal with the stack-switching done by the various | architectures to handle IRQ's, special-case exceptions, soft IRQ | handling, dealing with the assembly-code magic performed by the | entry.S entry points, and son on. I noted that your crash example | was a "clean" panic() call -- as opposed to a the far-more-likely | crash scenario where the kernel is running in kernel mode, takes | an exception, lays down an exception frame, switches stacks, etc... I'm not sure there is a significant problem there, Tom Reni at MontaVista wrote some kgdb enhancements for kgdb added Call Frame Instruction (CFI) macros that are added to assembler code for exception handlers to tell gdb how to deal with these assembler stack frames. I'm going to not only enable these for CONFIG_KGDB kernels but also for KEXEC kernels, as these are likely to generate crash dumps. I noticed your crash code has CFI support for stack unwinding. I more recent kernels may now support CFI already. I'm still on an old 2.6.12 kgdb patch. Jason Wessel is working and Andrew Morton to integrate the KGDB patch into the mainstream kernel. Last I read it looked like 2.6.25 was the likely target but I still don't see it in Linus's 2.6.25-rc5 git repo. Sigh! I'd like to see a consistent debugging environment from live debugging with kgdb over to crash analysis with gdb+ddd and crash. Likely the more crash is integrated into gdb the better. Perhaps over the long term integrating support into the kernel and gdb would be better. Remember the old -k option for dbx that Sun used when using dbxtool on the old M68000 kernels prior to SPARC? |> | > |> | > Also why don't you set up gdb with the stack info for the current |> | > task and either allow the user to do a 'gdb bt' or do it with a |> | > crash 'bt' option? |> | > |> | |> | I don't even come close to understanding gdb internals well enough |> | to know what "set up gdb with the stack info" entails. Send me |> | a patch when you've got it working -- for all arches, for all tasks, |> | and without requiring any vmcore file argument... ;-) What's the downside to providing the vmcore file to gdb when you start it and then using the internal command line interface to set thing like $sp and $fp to get gdb to do it's thing during a backtrace? Maybe I can get gdb on core files to allow me to change $sp and $fp with a minor gdb change. If that works I would think it easy to get crash to do the same to it's gdb pipe and if you provide the core to gdb on startup. Once gdb has the stack context, like gdb on old SunOS 4.1.4, then crash can pass thru cmds to walk up and down the stack. Am I dreaming? It tried just changing a global variable in the core file, starting ddd --write and it core dumped when I changed the global variable. |> | |> | > Will gdb work directly with the kdump core file? If not when did |> | > it break and why? Docs with 2.6.16 suggest to use gdb of the core |> | > file. |> | |> | I don't know if it *has* broke... |> | |> | As I said this morning, I don't know if the latest and greatest 32-bit |> | gdb can work with 64-bit ELF kdump vmcores, which kdump uses by default |> | for both 32-bit and 64-bit architectures. The kdump.txt file says |> this: |> | |> | * By default, the ELF headers are stored in ELF64 format to support |> | systems with more than 4GB memory. The --elf32-core-headers |> option can |> | be used to force the generation of ELF32 headers. This is necessary |> | because GDB currently cannot open vmcore files with ELF64 |> headers on |> | 32-bit systems. ELF32 headers can be used on non-PAE systems |> (that is, |> | less than 4GB of memory). |> | |> | But at least with a 32-bit ELF vmcore, gdb should be able to work. |> |> ok, I'll trying pulling out a DIMM and running on 2GB. | | But you'll still get a 64-bit vmcore unless you deal with the | kexec/kdump configuration. And why not just put "mem=2048m" on | the boot command line? Yea, that's when I did. The back traces for eight CPU's look fine, thought I haven't enabled the KGDB CFI macros yet. Data browsing our sockets was great. I showed the guys last night; reaction from one of our managers was "Very cool". - -piet