On Fri, May 05, 2006 at 08:59:04AM +0200, Michael Holzheu wrote: If I am not wrong you meant, that "lcrash" is able to show user space stack backtrace because the dump includes data from swap devices & files? Just wondering if you have used "crash" in the same manner? Maneesh

On Fri, May 05, 2006 at 09:34:39AM +0200, Michael Holzheu wrote: Thanks, I got it now. But IMHO this functionality more looks like part of crash dumping mechanism than crash analysis tool. In anycase it should be useful. Maneesh

Maneesh Soni wrote: Certainly a welcome task. I suggest segregating the code in a separate file (as done with lkcd_x86_trace.c), and the new entry point can simply be plugged into machdep->back_trace function pointer at init time. There should also be an "out" to allow it to be set back to use the current x86_64_low_budget_back_trace_cmd(). Also, if it doesn't support -fomit-frame-pointer, it's not worth doing. I don't quite understand how moving code to libraries is going to achieve the goal here. Things in some of the various *_init() functions could certainly be streamlined (or skipped) in order to make it more likely to make it to the first prompt. For example, the task table initialization could be made to simply fill in the context data for just the panic task. (But it almost sounds like you just want to use gdb alone for the minimal set of commands you've listed?) As far as "re-writes" are concerned, please keep in mind the necessity of backwards-compatibility. I'd much rather keep the current code -- that's known to work -- in place, and if you come up with something new, or re-shuffled, make it only callable when the kernel is of a known kernel version or later. The point is, let's not just re-invent the wheel just for purpose of re-inventing the wheel. I don't particularly like this suggestion. (I thought we just went through a problem where Ubuntu kernels don't even have kernel headers?) As far as code reuse, we already do that in a number of places, so I guess that's OK. And there is just never seems to be a "one-size-fits-all" set of kernel functions/macros that covers all bases over the life of the kernel and each processor type. But as always, I'm open to suggestion. Not a bad idea... OK. Often this gets recognized already, or if things are horribly corrupted, the session won't even come up. Have at it... Given the x86_64 usage of registers for passing args, good luck. Again, I guess it might be nice. Talk to gdb -- that's where it all comes from... For any text address, gdb has the associated line number data. It often looks confusing because the text comes from a header macro or inline or whatever. I don't know what you can do about that. Needs more of a description... I thought crash was a kernel [crash/live-system] analyzer? You currently can add user-space debug data with "add-symbol-file", which loads the debug data and symbols into gdb. I have done this kind of thing, but it's been an "almost-never" kind of situation, where I've wanted to display a user program's data structure. But if you want to start throwing in this kind of user-space stuff, please just keep it segregated. Not clear on what "establishing" CPU registers means. We already dump exception frames. I guess you mean to be able to use a register connotation in certain commands, as opposed to the address contained in the register? That's potentially messy, because it puts processor-specific stuff in processor-neutral code. As far as the LDT, GDB, task gates formatting, that's fine. No way -- we've been through this before. It is essentially a complete re-write. If you want this, make a new command entirely. I've made my personal feelings on these kinds of things before, which is to take a "minimalist" approach. Every new bell and whistle is virtually guaranteed to break as the kernel churns. And they all require an additional support burden. If I had my druthers, crash would have less rather than more at this point. But I understand that this has become a community project, and with the few exceptions above, I'm open to all patch suggestions. Thanks, Dave

Michael Holzheu wrote: But like you also mentioned, there's the extra burden of potentially having to save the swap partition contents someplace? That seems kind of ugly... Dave

Castor Fu wrote: Hey Castor, Nice to hear from an really old crash contributor! I mean in terms of crash's 7-year lifespan -- I don't know whether you're a boarder or a greybeard... Anyway, thinking about this gain, this kind of crash extension does seem reasonable. You've got all the information you need readily available; just doing a "vm -p" shows you where everything is. That is of course if kexec/kdump doesn't trash the current task's pgd. Although I wasn't aware that it even did so, this recent thread on the fastboot list highlighted that issue: Dave

crash-utility-bounces(a)redhat.com wrote on 05.05.2006 17:34:44: and Right! find Right, this is something, we from Linux for zSeries desperately need. In Böblingen we currently only use cross lcrash on an intel dump server, which can debug s390 and s390x dumps. This is an important reason, why crash currently is no option for us. language how Right, we have kernel areas, where we are dependent on the scripting support in lcrash. One example is our s390x common IO layer, where we could have hunderds of device structures, which you can't parse by hand. Here we use lcrash sial scripts to do the analysis. But I would really vote for a C-like language. This is definitely a big advantage of sial, since you can copy kernel code and data structures 1:1 into your analysis scripts! possible I also agree here! Especially, if your kernel crashes at a very early time during booting or big memory areas are damaged, this feature is very important! Michael

On Mon, 2006-05-08 10:58:39, Dave Anderson wrote: That strategy is helpful but for my team not as useful as you might think. We only have one production server in our lab. It is an x86 system with eight CPUs and 8 GiB of memory. Everything else in our lab is used for problem reproduction. So you never know exactly what state they're in. And most of the lab systems have only 1 GiB of memory which makes analyzing large dumps an exercise in thrashing the disks. Our little s390 system doesn't even have that much memory assigned to each LPAR. Also, we haven't been able to convince management to spend the money to upgrade our lab infrastructure. So we're limited to 100 Mbps ethernet between most of the systems. Heck, my desktop (dual core AMD 64 with 4 GiB of memory and two Western Digital Raptor SATA disks) is more powerful than almost all of my lab systems. As a consequence that is where I do all of my x86/x86_64 analysis. Dave, have you heard from rainer.bawidamann(a)de.ibm.com? He's leading an effort to create a cross-arch capable version of crash. Last I heard the current implementation only supports host=x86/target=ppc32. -- Kurtis D. Rader, Level 3 Linux Support ABC Service Center, Linux Change Team T/L 775-3714, DID +1 503-578-3714

Time for me to speak up I guess ... Prior on, system s390 upgrade We're on the other side of the range: embedded system with <64MB of Ram, no disk, just flash, not that fast. You don't want to run crash there. OTOH every laptop is fast and big enough for crash with these dumps so we debug our dumps on Thinkpads. Of course we could use crash native on any other Linux-PPC32 system, but the faster PPCs are 64bit and that's already cross-platform. That's right, but the patch is ugly and very intrusive, you won't like it. And it might be difficult to get x64/s390 eg. as some kernel headers from the target are needed for compiling crash. I plan to make the patch cleaner and less intrusive, but people keep putting other things on my plan. And I need to resolve some other tasks before I can send patches, so don't hold your breath. Regards, Rainer

On Fri, May 05, 2006 at 08:34:44AM -0700, Kurtis D. Rader wrote: Agreed, this is also necessary as an added advantage for the service people. gdb itself have some scripting support probably not as advanced as perl or python. Though I have never tried invoking gdb scripts via crash but I think it should be possible. I have anyway included this also in the new list. Agreed. But I think some platforms like ppc64 might behave better which always use registers for passing parameters. Thanks maneesh

Dave Anderson wrote: A big advantage to this approach is the ability to have multiple libraries, one per per kernel version, at minimum a crash library for each major kernel version 2.4, 2.6, 2.7.. This would preserve backward compatibility as we can leave all the 2.4 stuff alone when adding code for the 2.7 kernel for example. If a distro has special needs for crash they can provide a crash library specific to their kernel. If crash is designed so that some minimal analysis can be performed with out a dependency on the crash library (look at the running cpus back traces, read the log..) then a service person can get started on a crash dump and determine if a library will be needed or if changes to the library are needed for the specific kernel. Often you just need a little information from a dump to identify known problems in the kernel. -- David Wilder IBM Linux Technology Center Beaverton, Oregon, USA dwilder(a)us.ibm.com (503)578-3789

On Fri, May 05, 2006 at 10:03:54AM -0400, Dave Anderson wrote: ok, thanks for the suggestion. I have added this in the modified list appended below and request Rachita to keep this in mind. The main aim is to have crash atleast make it to the first prompt. And for advanced commands either we can try postponing *_init() function till the first invocation or keep them in libraries. Agreed, backward compatibility should be maintained. Actually final form of the solution is still not decided. It could be kernel headers, or some binary (library). There has been some patches regarding "make headers_install" which need some investigation also to see if they can be of some help here. <snip> Badari, I guess you meant some command to dump all per-cpu data for each cpu? or some specific data. Please correct me here. ok, I hope keeping all such commands in separate library should not be objectionable. Right. We can have commands to dump specific register contents or use registers as arguments to some commands. Probably we can use extended libraries for such command to reduce the clutter. Ok, but this looks like high on priority for some and low for some. In anycase this should be done in acceptable way. Thanks. I have appended the modified list below, keeping you suggestions in mind. Maneesh -------------------------------------------------------------------------------- DESCRIPTION: clean & correct stack back traces on platforms ALL the time. - x86_64 (currently wrong and need fixing) - segregate the code in a separate file (as done with lkcd_x86_trace.c), and the new entry point can simply be plugged into machdep->back_trace function pointer at init time. - There should also be an "out" to allow it to be set back to use the current x86_64_low_budget_back_trace_cmd(). Also, if it doesn't support -fomit-frame-pointer, it's not worth doing. RESOLUTION STATUS: Work-in-progress by Rachita Kothiyal <rachita(a)in.ibm.com&gt; -------------------------------------------------------------------------------- DESCRIPTION: Code restructuring: - streamline the *_init() functions so as that crash is at least able to open the dump image and perform minimal set of commands like bt, dump dmesg log, disassemble etc. irrespective of kernel version. - code is hard to read & understand - need to re-write some of the basic subsystems like memory mapping, pagetable management etc maintaining the backward compatibility. RESOLUTION STATUS: Work-in-progress by Dave Wilder <dwilder(a)us.ibm.com&gt; and Maneesh Soni <maneesh(a)in.ibm.com&gt; -------------------------------------------------------------------------------- DESCRIPTION: Crash & kernel version independence: - kernel headers & code - reuse ? It would be nice to figure out a way to include kernel headers and sections of kernel code to do hard stuff (like memory mapping functions page_to_pfn, pfn_to_page, pagetable decoding etc..). RESOLUTION STATUS: Work-in-progress by Dave Wilder <dwilder(a)us.ibm.com&gt; and Maneesh Soni <maneesh(a)in.ibm.com&gt; -------------------------------------------------------------------------------- DESCRIPTION: Mini report: - The goal of this is to produce a summary report of common information that is used to track problems. The idea here is for many problems we probably don't need to get the whole dump shipped and as you probably figured out by now it is not easy to ship and store these huge dump files. RESOLUTION STATUS: TBD -------------------------------------------------------------------------------- DESCRIPTION: Automatic verification of the dump: - When you get a dump to look at problem there are few common tasks one performs, the idea here is to automate those tasks and provide a simple interface in the tool. Another possibility is automatic verification of important datastructures, for example if the task list says there are 30 tasks this feature automatically walks the list and counts to verify if there are 30 in the list or not, if 30 entries or not found this may give a clue of some kind of a corruption. RESOLUTION STATUS: TBD -------------------------------------------------------------------------------- DESCRIPTION: function arguments: - Display arguments in the stack trace. At present, we do not have support for PPC64 and x86_64. On PPC64, user can dump retrieve only for top level frame from pt_regs. However, user can dump complete stack frame and read arguments. So, it is manual process and need to have some expertise on the stack frame RESOLUTION STATUS: TBD -------------------------------------------------------------------------------- DESCRIPTION: local variables: - Facilitate possible display of local variables with stack frames Since we are using debug vmlinux, we can find local variables locations from Dwarf2. RESOLUTION STATUS: TBD -------------------------------------------------------------------------------- DESCRIPTION: better assembly & source languge, line# display in disassembly - interacting with gdb might help as for any text address, gdb has the associated line number data but there might be some confusion depending up the source of text. RESOLUTION STATUS: TBD -------------------------------------------------------------------------------- DESCRIPTION: per-cpu info (like stacks traces) - Display all or specific per cpu data for all cpus or specific cpu. RESOLUTION STATUS: TBD -------------------------------------------------------------------------------- DESCRIPTION: User space enhancements - show user space stack backtrace, if present in the dump file, - ability to link user space namelist (debug object files), RESOLUTION STATUS: TBD -------------------------------------------------------------------------------- DESCRIPTION: Platform specific enhancements - Establish CPU registers at the time of exceptions in the current context - Ability to handle CPU registers from current context using symbols in expressions - Ability to format basic processor structures like LDT, GDT, task gates for x86 arch RESOLUTION STATUS: TBD -------------------------------------------------------------------------------- DESCRIPTION: cross architecture support for crash RESOLUTION STATUS: TBD -------------------------------------------------------------------------------- DESCRIPTION: scripting support - integrating scripting support with perl or python like language, "Alicia" can be one example or the solution itself. RESOLUTION STATUS: TBD