Re: [Crash-utility] crashdc : automatic data collection for new vmcores in text format
by Dave Anderson
----- "Louis Bouchard" <louis.bouchard(a)hp.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello,
>
> > Is /usr/bin/crashdc itself a shell script?
> >
> > Dave
>
> Yes, crashdc itself is a shell script, as well as the run-crashdc-*
> which are intended to be invoked by the kdump mechanisms (kdump_post in
> RHEL, KDUMP_POSTSCRIPT in SLES).
I'm guessing the run-*.sh scripts are easy to modify in order for
a user to tailor them to their needs?
Anyway, it sounds reasonable to me.
Can you work up a patch to the crash.spec file to create an additional
sub-package? Presume a new crashdc subdirectory off the top-level
to contain the files so that tar.gz users will get them as well.
Thanks,
Dave
15 years
crashdc : automatic data collection for new vmcores in text format
by Louis Bouchard
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello everyone.
In a previous life, I was doing Tru64 support and we had a crashdc
script that would collect result of a small set of data out of the crash
dump to a text file every time a new dump would get created. The
advantage of this is to allow a customer to send the basic dump
information in a text format to support engineers for a first pass.
Dumps are getting big these days and it can take days to get the whole
vmcore file.
So I took the idea and recreated the mechanism on linux based on crash.
It's a small shell script that build a crash command file and feeds it
to the crash utility automatically. It can also be invoked manually on
existing dumps rather easily. Here is its format :
$ ./crashdc -h
usage: crashdc [-d level] [-h] [-V] [-x /path/to/crash/exe]
/path/to/namelist /path/to/vmcore /path/to/output/directory
I'm relying on existing kdump mechanism to automate the collection of
data whenever a new vmcore is generated. Right now, I have it working
on RHEL5, close to be working on SLES11 and not working on SLES10. But
this is not why I'm telling you all this.
I am coming to you to know if there would be some interest in including
those scripts into the crash-utility rpm. I'm ready to package it
myself and to give those scripts their own home, but since the
unavoidable dependency is to have crash present, I was thinking that it
might be better to have it included with crash.
Right now, here is what is included :
- /usr/bin/crashdc
- /usr/bin/run-crashdc-rhel.sh
- /usr/bin/run-crashdc-sles.sh
- /usr/bin/run-crashdc-sles11.sh
- /usr/share/man/man8/crashdc.8
I would like to have your opinion and comments on this suggestion, and
on the value of having such a script available for general use.
Kind Regards,
- --
Louis Bouchard, Linux Support Engineer
Team lead, EMEA Linux Competency Center,
Linux Ambassador, HP
HP Services 1 Ave du Canada
HP France Z.A. de Courtaboeuf
louis.bouchard(a)hp.com 91 947 Les Ulis
http://www.hp.com/go/linux France
http://www.hp.com/fr
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkq8yHIACgkQDvqokHrhnCxmhgCfWuoNlw3idNdesOP8A4wEA8T8
5tEAoO/JPQlMctXPMn5s+NyzawGXFs39
=DmX/
-----END PGP SIGNATURE-----
15 years
Re: Fix for source line numbers for x86_64 modules
by Dave Anderson
----- "John Wright (ALPS, Fort Collins)" <john.wright(a)hp.com> wrote:
> > crash> dis -l register_kprobes
> > < never returns >
>
> I wonder if it would eventually return after minutes or hours? (Not
> that this is acceptable - I'm just curious.)
Anyway, I tried it again, and it came back 6 minutes later,
so my "never returns" claim was a bit of a exaggeration...
Sorry about that...
But stranger still -- I mis-typed the hanging command above.
It's takes several minutes to run "dis -l register_kprobe", whereas
"dis -l register_kprobes" works OK -- they're both in the same
neighborhood:
ffffffff813af401 (T) register_kprobe
ffffffff813afac4 (T) register_kprobes
> > With your new patch (and unpatched for that matter) it works OK.
>
> Great! Sorry for the original mess. It was quick and dirty, and solved
> the specific problem Bob and I were having. I think this version is
> correct, though.
Queued for the next release.
Again, thanks very much,
Dave
15 years
Re: Fix for source line numbers for x86_64 modules
by Dave Anderson
----- "John Wright (ALPS, Fort Collins)" <john.wright(a)hp.com> wrote:
> On Wed, Sep 16, 2009 at 04:44:32PM -0600, Bob Montgomery wrote:
> > John and I think that this code in gdb searches things too many times,
> > particularly with this patch, but it's a start since it seems to fix the
> > problem.
>
> I'm attaching a new version of the patch, that performs way better when
> disassembling functions that live in the kernel. (Bob found that the
> original patch made crash disassemble in-kernel functions at least 3
> times slower, but that number will be larger depending on how close the
> symbol table the function lives in is to the head of the psymtabs list.
> Module disassembly speed wasn't changed much at all.) With this updated
> patch, we found the performance penalty of "dis -l" to be marginal.
>
> The problem with the original patch is that once the address we want is
> found in a symbol table, it then looks through the rest of the symbol
> tables in that objfile for a better match. The original code would then
> return the best pst out of that objfile (and never get the next pst
> from ALL_PSYMTABS), but we want to go through the rest of the objfiles
> just in case, so I moved the return statement outside of the
> ALL_PSYMTABS loop. But the next pst from ALL_PSYMTABS will not be from
> a new objfile - so we would wind up traversing the list (minus one
> element) again, and again, and again...
>
> The new patch removes the inner list traversal, and just takes advantage
> of the fact that we already iterate through every pst via
> ALL_PSYMTABS.
Hi John
Yeah, this second patch works much better. In fact, I only noticed today -- using
your first patch -- that if I pick a text address in the kernel proper that
is very close to "_etext", the disassembly never returns from gdb.
For example, on an 2.6.29.4-167.fc11 kernel, if I do this with the
first patch applied:
crash> sym -l
... [ snip ] ...
ffffffff813afac4 (T) register_kprobes
ffffffff813afb26 (T) recycle_rp_inst
ffffffff813afbbb (T) kprobe_flush_task
ffffffff813afc75 (t) collect_one_slot
ffffffff813afd18 (t) collect_garbage_slots
ffffffff813afda9 (T) free_insn_slot
ffffffff813afe4d (T) get_insn_slot
ffffffff813aff85 (T) __kprobes_text_end
ffffffff813b02ec (t) bad_iret
ffffffff813b0313 (t) bad_gs
ffffffff813b2250 (T) bad_from_user
ffffffff813b2256 (t) bad_to_user
ffffffff813b2830 (T) __start_notes
ffffffff813b2830 (T) _etext
...
crash> dis -l register_kprobes
< never returns >
With your new patch (and unpatched for that matter) it works OK.
Dave
15 years
Re: Fix for source line numbers for x86_64 modules
by Dave Anderson
----- "John Wright (ALPS, Fort Collins)" <john.wright(a)hp.com> wrote:
> On Wed, Sep 16, 2009 at 04:44:32PM -0600, Bob Montgomery wrote:
> > John and I think that this code in gdb searches things too many times,
> > particularly with this patch, but it's a start since it seems to fix the
> > problem.
>
> I'm attaching a new version of the patch, that performs way better when
> disassembling functions that live in the kernel. (Bob found that the
> original patch made crash disassemble in-kernel functions at least 3
> times slower, but that number will be larger depending on how close the
> symbol table the function lives in is to the head of the psymtabs list.
> Module disassembly speed wasn't changed much at all.) With this updated
> patch, we found the performance penalty of "dis -l" to be marginal.
>
> The problem with the original patch is that once the address we want is
> found in a symbol table, it then looks through the rest of the symbol
> tables in that objfile for a better match. The original code would then
> return the best pst out of that objfile (and never get the next pst
> from ALL_PSYMTABS), but we want to go through the rest of the objfiles
> just in case, so I moved the return statement outside of the
> ALL_PSYMTABS loop. But the next pst from ALL_PSYMTABS will not be from
> a new objfile - so we would wind up traversing the list (minus one
> element) again, and again, and again...
>
> The new patch removes the inner list traversal, and just takes advantage
> of the fact that we already iterate through every pst via ALL_PSYMTABS.
I thought I saw a bit of a hitch at times with the last patch.
I'll pull it and give this one a go...
Thanks,
Dave
15 years
Fix for source line numbers for x86_64 modules
by Bob Montgomery
This patch allows the dis -l command to show real source line numbers
for module code, instead of this sort of thing:
crash-4.0.9> dis -l bnx2_poll_msix
include/linux/cpumask.h: 612
0xffffffffa008acc3 <bnx2_poll_msix>: push %rbp
0xffffffffa008acc4 <bnx2_poll_msix+1>: mov %rsp,%rbp
0xffffffffa008acc7 <bnx2_poll_msix+4>: push %r15
0xffffffffa008acc9 <bnx2_poll_msix+6>: push %r14
0xffffffffa008accb <bnx2_poll_msix+8>: push %r13
0xffffffffa008accd <bnx2_poll_msix+10>: mov %esi,%r13d
0xffffffffa008acd0 <bnx2_poll_msix+13>: push %r12
0xffffffffa008acd2 <bnx2_poll_msix+15>: xor %r12d,%r12d
0xffffffffa008acd5 <bnx2_poll_msix+18>: push %rbx
0xffffffffa008acd6 <bnx2_poll_msix+19>: mov %rdi,%rbx
0xffffffffa008acd9 <bnx2_poll_msix+22>: sub $0x8,%rsp
The problem has to do with a symbol table in the kernel containing an
abnormally huge text range because of a goofy vsyscall reference in the
file "arch/x86/kernel/hpet.c".
(gdb) p *(struct block *)0x8d664b0
$41 = {startaddr = 0xffffffff80225130, endaddr = 0xffffffffff6001b3,
This range pretty much covers the entire module address space and since
the kernel file is first in the object_files list, the search stops
there and returns the closest symbol from the kernel, instead of looking
on through the object list to the module of actual interest.
This patch causes the code to keep looking through all modules and
return the real best symbol thingy (you can tell how much I really know
about gdb's symtab stuff, ahem). Thanks (and blame) to John Wright for
suggesting the fix once I found the problem.
The patch applies to the gdb patch file in the crash directory.
John and I think that this code in gdb searches things too many times,
particularly with this patch, but it's a start since it seems to fix the
problem. Mayhaps some gdb experts can now help do it right.
With the patch, the above example becomes:
crash-4.0.9> dis -l bnx2_poll_msix
/build/buildd/linux-2.6-clim-2.6.29-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 3215
0xffffffffa008acc3 <bnx2_poll_msix>: push %rbp
0xffffffffa008acc4 <bnx2_poll_msix+1>: mov %rsp,%rbp
0xffffffffa008acc7 <bnx2_poll_msix+4>: push %r15
0xffffffffa008acc9 <bnx2_poll_msix+6>: push %r14
0xffffffffa008accb <bnx2_poll_msix+8>: push %r13
0xffffffffa008accd <bnx2_poll_msix+10>: mov %esi,%r13d
0xffffffffa008acd0 <bnx2_poll_msix+13>: push %r12
/build/buildd/linux-2.6-clim-2.6.29-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 3219
0xffffffffa008acd2 <bnx2_poll_msix+15>: xor %r12d,%r12d
/build/buildd/linux-2.6-clim-2.6.29-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 3215
0xffffffffa008acd5 <bnx2_poll_msix+18>: push %rbx
0xffffffffa008acd6 <bnx2_poll_msix+19>: mov %rdi,%rbx
0xffffffffa008acd9 <bnx2_poll_msix+22>: sub $0x8,%rsp
Bob Montgomery
15 years
Re: Fix for source line numbers for x86_64 modules
by Dave Anderson
----- "Bob Montgomery" <bob.montgomery(a)hp.com> wrote:
> This patch allows the dis -l command to show real source line numbers
> for module code, instead of this sort of thing:
>
> crash-4.0.9> dis -l bnx2_poll_msix
> include/linux/cpumask.h: 612
> 0xffffffffa008acc3 <bnx2_poll_msix>: push %rbp
> 0xffffffffa008acc4 <bnx2_poll_msix+1>: mov %rsp,%rbp
> 0xffffffffa008acc7 <bnx2_poll_msix+4>: push %r15
> 0xffffffffa008acc9 <bnx2_poll_msix+6>: push %r14
> 0xffffffffa008accb <bnx2_poll_msix+8>: push %r13
> 0xffffffffa008accd <bnx2_poll_msix+10>: mov %esi,%r13d
> 0xffffffffa008acd0 <bnx2_poll_msix+13>: push %r12
> 0xffffffffa008acd2 <bnx2_poll_msix+15>: xor %r12d,%r12d
> 0xffffffffa008acd5 <bnx2_poll_msix+18>: push %rbx
> 0xffffffffa008acd6 <bnx2_poll_msix+19>: mov %rdi,%rbx
> 0xffffffffa008acd9 <bnx2_poll_msix+22>: sub $0x8,%rsp
>
> The problem has to do with a symbol table in the kernel containing an
> abnormally huge text range because of a goofy vsyscall reference in the
> file "arch/x86/kernel/hpet.c".
>
> (gdb) p *(struct block *)0x8d664b0
> $41 = {startaddr = 0xffffffff80225130, endaddr = 0xffffffffff6001b3,
>
> This range pretty much covers the entire module address space and since
> the kernel file is first in the object_files list, the search stops
> there and returns the closest symbol from the kernel, instead of looking
> on through the object list to the module of actual interest.
>
> This patch causes the code to keep looking through all modules and
> return the real best symbol thingy (you can tell how much I really know
> about gdb's symtab stuff, ahem). Thanks (and blame) to John Wright for
> suggesting the fix once I found the problem.
>
> The patch applies to the gdb patch file in the crash directory.
>
> John and I think that this code in gdb searches things too many times,
> particularly with this patch, but it's a start since it seems to fix the
> problem. Mayhaps some gdb experts can now help do it right.
>
> With the patch, the above example becomes:
>
> crash-4.0.9> dis -l bnx2_poll_msix
> /build/buildd/linux-2.6-clim-2.6.29-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 3215
> 0xffffffffa008acc3 <bnx2_poll_msix>: push %rbp
> 0xffffffffa008acc4 <bnx2_poll_msix+1>: mov %rsp,%rbp
> 0xffffffffa008acc7 <bnx2_poll_msix+4>: push %r15
> 0xffffffffa008acc9 <bnx2_poll_msix+6>: push %r14
> 0xffffffffa008accb <bnx2_poll_msix+8>: push %r13
> 0xffffffffa008accd <bnx2_poll_msix+10>: mov %esi,%r13d
> 0xffffffffa008acd0 <bnx2_poll_msix+13>: push %r12
> /build/buildd/linux-2.6-clim-2.6.29-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 3219
> 0xffffffffa008acd2 <bnx2_poll_msix+15>: xor %r12d,%r12d
> /build/buildd/linux-2.6-clim-2.6.29-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 3215
> 0xffffffffa008acd5 <bnx2_poll_msix+18>: push %rbx
> 0xffffffffa008acd6 <bnx2_poll_msix+19>: mov %rdi,%rbx
> 0xffffffffa008acd9 <bnx2_poll_msix+22>: sub $0x8,%rsp
Outstanding!
BTW, I don't know if there *are* any gdb experts on this list...
If there are, I hope they can speak up!
I haven't tested this patch or taken the time to understand it,
but I'm taking it on good faith that it solves this bug, which
has been an elephant in the room since (I believe) the 2.6.21
timeframe.
In fact, Mike Snitzer had narrowed it down to the kernel patch that
seemingly caused it, and you've confirmed it:
https://www.redhat.com/archives/crash-utility/2009-January/msg00063.html
which was this one:
commit 7460ed2844ffad7141e30271c0c3da8336e66014
Author: john stultz <johnstul(a)us.ibm.com>
Date: Fri Feb 16 01:28:21 2007 -0800
[PATCH] time: x86_64: re-enable vsyscall support for x86_64
Cleanup and re-enable vsyscall gettimeofday using the generic clocksource
infrastructure.
[akpm osdl org: cleanup]
Signed-off-by: John Stultz <johnstul us ibm com>
Cc: Ingo Molnar <mingo elte hu>
Cc: Thomas Gleixner <tglx linutronix de>
Cc: Andi Kleen <ak muc de>
Cc: Roman Zippel <zippel linux-m68k org>
Signed-off-by: Andrew Morton <akpm linux-foundation org>
Signed-off-by: Linus Torvalds <torvalds linux-foundation org>
arch/x86_64/Kconfig | 4 +
arch/x86_64/kernel/hpet.c | 6 ++
arch/x86_64/kernel/time.c | 6 --
arch/x86_64/kernel/tsc.c | 7 ++
arch/x86_64/kernel/vmlinux.lds.S | 28 ++++------
arch/x86_64/kernel/vsyscall.c | 119 ++++++++++++++++++++++---------------
include/asm-x86_64/proto.h | 2 -
include/asm-x86_64/timex.h | 1 -
include/asm-x86_64/vsyscall.h | 29 ++--------
9 files changed, 104 insertions(+), 98 deletions(-)
Anyway, this is awesome -- I (and countless others) really appreciate it...
Thanks to you and John,
Dave
15 years
[PATCH] sial: Fix processing of bitfields on big endian systems
by Michael Holzheu
Hi Luc,
The processing of bit fields on big endian systems in sial is currently
broken, because the bits are not copied to the correct position and are
not shifted the right way.
To fix this, the following patch does the processing on big endian
systems as follows:
1. sial_exemem(): Copy complete bit field to the "end" (right) of the
long long variable.
2. get_bit_value(): Shift the bits of the bit field member right to
the "end" of the long long variable. This results in the value of
the requested bitfield member.
Michael
---
extensions/libsial/sial_member.c | 7 +++++--
extensions/libsial/sial_type.c | 6 +++++-
2 files changed, 10 insertions(+), 3 deletions(-)
Index: crash-4.0.9/extensions/libsial/sial_member.c
===================================================================
--- crash-4.0.9.orig/extensions/libsial/sial_member.c
+++ crash-4.0.9/extensions/libsial/sial_member.c
@@ -236,10 +236,13 @@ srcpos_t p;
}
/* bit field gymnastic */
else if(stm->m.nbits) {
-
ull value=0;
+ void *target = &value;
+
+ if (__BYTE_ORDER != __LITTLE_ENDIAN)
+ target = target + (sizeof(value) - stm->m.size);
- API_GETMEM(m->mem+stm->m.offset, &value, stm->m.size);
+ API_GETMEM(m->mem+stm->m.offset, target, stm->m.size);
get_bit_value(value, stm->m.nbits, stm->m.fbit, stm->m.size, v);
/* no mempos for bit fields ... */
Index: crash-4.0.9/extensions/libsial/sial_type.c
===================================================================
--- crash-4.0.9.orig/extensions/libsial/sial_type.c
+++ crash-4.0.9/extensions/libsial/sial_type.c
@@ -287,7 +287,11 @@ get_bit_value(ull val, int nbits, int bo
else {
mask = ((1 << nbits) - 1);
}
- val = val >> boff;
+
+ if (__BYTE_ORDER == __LITTLE_ENDIAN)
+ val = val >> boff;
+ else
+ val = val >> (vnbits - boff - nbits);
val &= mask;
if(issigned(v)) {
15 years
Re: Redirect with crash gdb commands
by Dave Anderson
----- "Bob Montgomery" <bob.montgomery(a)hp.com> wrote:
> I really like to use the x command in crash, as in:
>
> crash-4.0.9> x/256xg 0xffff8801aa9c8000
> 0xffff8801aa9c8000: 0xffff8800c61541c0 0xffff880145a194c0
> 0xffff8801aa9c8010: 0xffff88016fa481c0 0xffff88016c7980c0
> ...
>
>
> I like it better than the rd command (because of prior familiarity
> perhaps?):
> crash-4.0.9> rd -x -64 0xffff8801aa9c8000 256
> ffff8801aa9c8000: ffff8800c61541c0 ffff880145a194c0
> ffff8801aa9c8010: ffff88016fa481c0 ffff88016c7980c0
> ...
BTW, you don't need the "-x", the "-64", or the "0x" -- given that they are
the defaults. So "rd" beats "x" 21 to 25 in keystrokes required... ;-)
>
> But I *really* like crash's ability to pipe and redirect commands, but
> that doesn't work with the x command:
>
> crash-4.0.9> x/256xg 0xffff8801aa9c8000 | grep ffff880145a194c0
> crash-4.0.9>
>
> But it works with rd:
>
> crash-4.0.9> rd -x -64 0xffff8801aa9c8000 256 | grep ffff880145a194c0
> ffff8801aa9c8000: ffff8800c61541c0 ffff880145a194c0
> crash-4.0.9>
>
> Another fun one:
> crash-4.0.9> x/256xg 0xffff8801aa9c8000 | head
> Argument to arithmetic operation not a number or boolean.
>
>
> If you redirect the output of x, you get a file containing an error
> message:
>
> crash-4.0.9> x/256xg 0xffff8801aa9c8000 >xold.out
> crash-4.0.9> !sh
> sh-3.2$ cat xold.out
> No symbol "xold" in current context.
> sh-3.2$
>
> The problem is that the setup of gdb commands in is_gdb_command() with
> merge_orig_args set puts the "| thing" or "> thing" back onto the
> command it is building to be passed through to gdb.
>
> I first thought that cmd_gdb() passing a NULL, instead of the fp with
> the redirection set up, into gdb_pass_through was part of the problem
> also, but that seems to be dealt with later in gdb_interface.
>
> There might some other reason for the strange behavior of
> merge_orig_args that my testing has not found. But otherwise, the
> attached patch will allow my favorite x command to be sent through pipes
> and redirected to files.
Nice.
What your patch does is to accurately mimic what happens when you
precede your command with "gdb", as in:
crash> gdb x/256xg 0xffff8801aa9c8000 > xold.out
The syntax above works with crash as it is now, but your patch allows
it to work without having to prepend the "gdb". I never noticed
that anomoly before, probably because I rarely use gdb commands, and
when I do, I usually don't redirect them.
Queued for the next release.
Thanks,
Dave
15 years
Redirect with crash gdb commands
by Bob Montgomery
I really like to use the x command in crash, as in:
crash-4.0.9> x/256xg 0xffff8801aa9c8000
0xffff8801aa9c8000: 0xffff8800c61541c0 0xffff880145a194c0
0xffff8801aa9c8010: 0xffff88016fa481c0 0xffff88016c7980c0
...
I like it better than the rd command (because of prior familiarity
perhaps?):
crash-4.0.9> rd -x -64 0xffff8801aa9c8000 256
ffff8801aa9c8000: ffff8800c61541c0 ffff880145a194c0
ffff8801aa9c8010: ffff88016fa481c0 ffff88016c7980c0
...
But I *really* like crash's ability to pipe and redirect commands, but
that doesn't work with the x command:
crash-4.0.9> x/256xg 0xffff8801aa9c8000 | grep ffff880145a194c0
crash-4.0.9>
But it works with rd:
crash-4.0.9> rd -x -64 0xffff8801aa9c8000 256 | grep ffff880145a194c0
ffff8801aa9c8000: ffff8800c61541c0 ffff880145a194c0
crash-4.0.9>
Another fun one:
crash-4.0.9> x/256xg 0xffff8801aa9c8000 | head
Argument to arithmetic operation not a number or boolean.
If you redirect the output of x, you get a file containing an error
message:
crash-4.0.9> x/256xg 0xffff8801aa9c8000 >xold.out
crash-4.0.9> !sh
sh-3.2$ cat xold.out
No symbol "xold" in current context.
sh-3.2$
The problem is that the setup of gdb commands in is_gdb_command() with
merge_orig_args set puts the "| thing" or "> thing" back onto the
command it is building to be passed through to gdb.
I first thought that cmd_gdb() passing a NULL, instead of the fp with
the redirection set up, into gdb_pass_through was part of the problem
also, but that seems to be dealt with later in gdb_interface.
There might some other reason for the strange behavior of
merge_orig_args that my testing has not found. But otherwise, the
attached patch will allow my favorite x command to be sent through pipes
and redirected to files.
Thanks,
Bob Montgomery
15 years