3:02 a.m.
Hello,
We wish to call 'crash' with a script file to automate some basic
command and output the result on a text file.
This works perfectly for basic commands such as bt, log, mod and so on,
especially when no input data is required.
We try now to dump the kernel stack through a 'rd <@> <size>'.
My problem is to "extract" this <@> and to "input" it in this
'rd' command.
For the moment, I'm able to get the task_struct * with following command :
ps | grep '>' | awk '{ print $5 };'
From that, the pointer of the thread_info would be available through
(task_struct *)->stack.
But how to get all these together to finally ouptut a valid 'rd' command ?
Meaby another way to get this kernel stack dump would exist ?
Any help would be appreciate.
Regards,
Patrick Agrain
3:23 a.m.
Bonjour Patrick,
Le 06/12/2013 10:02, Patrick Agrain a écrit :
Hello,
We wish to call 'crash' with a script file to automate some basic command and
output the result on a text file.
This works perfectly for basic commands such as bt, log, mod and so on,
especially when no input data is required.
We try now to dump the kernel stack through a 'rd <@> <size>'.
My problem is to "extract" this <@> and to "input" it in this
'rd' command.
For the moment, I'm able to get the task_struct * with following command :
ps | grep '>' | awk '{ print $5 };'
From that, the pointer of the thread_info would be available through
(task_struct *)->stack.
But how to get all these together to finally ouptut a valid 'rd' command ?
Meaby another way to get this kernel stack dump would exist ?
Any help would be appreciate.
Regards,
Patrick Agrain
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
I did that kind of work a few years ago as the crashdc project :
http://crashdc.sourceforge.net/
The project is now dormant (elegant way of saying abandoned) but everything in
there is still valid, so you might be able to reuse some of it.
One of the most efficient way I found to work on specific address is to output
to a file from within crash, work a list of command from that file output and
feed back crash with this list.
A quick example would go like that. If you want active.mm.mm_count for each task
you could do :
crash> foreach task | grep active_mm | awk '{print "* mm_struct.mm_count
",$3}'
tempfile
crash> head -1 tempfile
* mm_struct.mm_count 0xffff880c46af9880,
crash> < tempfile
crash> * mm_struct.mm_count 0xffff880c46af9880,
mm_count = {
counter = 0x4
}
...
I'd be happy to get suggestions from others as well.
Kind regards,
...Louis
--
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu support: http://canonical.com/support
3:50 a.m.
Hi Patrick,
You can use the combination of foreach, awk and cut commands to get the
desired results.
Eg:
crash> foreach RU task|grep " stack =" |awk '{print $3}'|cut -f1
-d"," |awk
'{print "rd "$1 " " 3 }' > rd_stack_number
crash> rd 0xffffffff81a00000 3
ffffffff81a00000: ffffffff81a8d020 ffffffff81a9a8a0 ...............
ffffffff81a00010: 0000000400000000 ........
crash> rd 0xffff88106961e000 3
ffff88106961e000: ffff884069dd8040 ffffffff81a9a8a0 @..i@...........
ffff88106961e010: 0000000400000000 ........
The above example is some what similar to the Louis example.
-- BKS
On 6 December 2013 14:32, Patrick Agrain
<patrick.agrain(a)alcatel-lucent.com>wrote:
Hello,
We wish to call 'crash' with a script file to automate some basic command
and output the result on a text file.
This works perfectly for basic commands such as bt, log, mod and so on,
especially when no input data is required.
We try now to dump the kernel stack through a 'rd <@> <size>'.
My problem is to "extract" this <@> and to "input" it in this
'rd' command.
For the moment, I'm able to get the task_struct * with following command :
ps | grep '>' | awk '{ print $5 };'
From that, the pointer of the thread_info would be available through
(task_struct *)->stack.
But how to get all these together to finally ouptut a valid 'rd' command ?
Meaby another way to get this kernel stack dump would exist ?
Any help would be appreciate.
Regards,
Patrick Agrain
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
BKS
3:52 a.m.
Forgot to mentioned the following step; ;)
crash> < rd_stack_number |head -n 6
crash> rd 0xffffffff81a00000 3
ffffffff81a00000: ffffffff81a8d020 ffffffff81a9a8a0 ...............
ffffffff81a00010: 0000000400000000 ........
crash> rd 0xffff88106961e000 3
ffff88106961e000: ffff884069dd8040 ffffffff81a9a8a0 @..i@...........
ffff88106961e010: 0000000400000000 ........
On 6 December 2013 15:20, Buland Kumar Singh <6b65726e656c(a)gmail.com> wrote:
Hi Patrick,
You can use the combination of foreach, awk and cut commands to get the
desired results.
Eg:
crash> foreach RU task|grep " stack =" |awk '{print $3}'|cut -f1
-d","
|awk '{print "rd "$1 " " 3 }' > rd_stack_number
crash> rd 0xffffffff81a00000 3
ffffffff81a00000: ffffffff81a8d020 ffffffff81a9a8a0 ...............
ffffffff81a00010: 0000000400000000 ........
crash> rd 0xffff88106961e000 3
ffff88106961e000: ffff884069dd8040 ffffffff81a9a8a0 @..i@...........
ffff88106961e010: 0000000400000000 ........
The above example is some what similar to the Louis example.
-- BKS
On 6 December 2013 14:32, Patrick Agrain <
patrick.agrain(a)alcatel-lucent.com> wrote:
> Hello,
>
> We wish to call 'crash' with a script file to automate some basic command
> and output the result on a text file.
>
> This works perfectly for basic commands such as bt, log, mod and so on,
> especially when no input data is required.
>
> We try now to dump the kernel stack through a 'rd <@> <size>'.
> My problem is to "extract" this <@> and to "input" it in
this 'rd'
> command.
>
> For the moment, I'm able to get the task_struct * with following command :
> ps | grep '>' | awk '{ print $5 };'
>
> From that, the pointer of the thread_info would be available through
> (task_struct *)->stack.
>
> But how to get all these together to finally ouptut a valid 'rd' command ?
> Meaby another way to get this kernel stack dump would exist ?
>
> Any help would be appreciate.
> Regards,
> Patrick Agrain
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
--
BKS
--
BKS
4:04 a.m.
Thanks to all,
I'll give it a try. Looks good.
Regards,
Patrick Agrain
Le 06/12/2013 10:52, Buland Kumar Singh a écrit :
Forgot to mentioned the following step; ;)
crash> < rd_stack_number |head -n 6
crash> rd 0xffffffff81a00000 3
ffffffff81a00000: ffffffff81a8d020 ffffffff81a9a8a0 ...............
ffffffff81a00010: 0000000400000000 ........
crash> rd 0xffff88106961e000 3
ffff88106961e000: ffff884069dd8040 ffffffff81a9a8a0 @..i@...........
ffff88106961e010: 0000000400000000 ........
On 6 December 2013 15:20, Buland Kumar Singh <6b65726e656c(a)gmail.com
<mailto:6b65726e656c@gmail.com>> wrote:
Hi Patrick,
You can use the combination of foreach, awk and cut commands to
get the
desired results.
Eg:
crash> foreach RU task|grep " stack =" |awk '{print $3}'|cut
-f1
-d"," |awk '{print "rd "$1 " " 3 }' >
rd_stack_number
crash> rd 0xffffffff81a00000 3
ffffffff81a00000: ffffffff81a8d020 ffffffff81a9a8a0 ...............
ffffffff81a00010: 0000000400000000 ........
crash> rd 0xffff88106961e000 3
ffff88106961e000: ffff884069dd8040 ffffffff81a9a8a0 @..i@...........
ffff88106961e010: 0000000400000000 ........
The above example is some what similar to the Louis example.
-- BKS
On 6 December 2013 14:32, Patrick Agrain
<patrick.agrain(a)alcatel-lucent.com
<mailto:patrick.agrain@alcatel-lucent.com>> wrote:
Hello,
We wish to call 'crash' with a script file to automate some
basic command and output the result on a text file.
This works perfectly for basic commands such as bt, log, mod
and so on, especially when no input data is required.
We try now to dump the kernel stack through a 'rd <@>
<size>'.
My problem is to "extract" this <@> and to "input" it
in this
'rd' command.
For the moment, I'm able to get the task_struct * with
following command :
ps | grep '>' | awk '{ print $5 };'
>From that, the pointer of the thread_info would be available
through (task_struct *)->stack.
But how to get all these together to finally ouptut a valid
'rd' command ?
Meaby another way to get this kernel stack dump would exist ?
Any help would be appreciate.
Regards,
Patrick Agrain
--
Crash-utility mailing list
Crash-utility(a)redhat.com <mailto:Crash-utility@redhat.com>
https://www.redhat.com/mailman/listinfo/crash-utility
--
BKS
--
BKS
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
7:57 a.m.
----- Original Message -----
Hello,
We wish to call 'crash' with a script file to automate some basic
command and output the result on a text file.
This works perfectly for basic commands such as bt, log, mod and so on,
especially when no input data is required.
We try now to dump the kernel stack through a 'rd <@> <size>'.
My problem is to "extract" this <@> and to "input" it in this
'rd' command.
You can dump the full kernel stack with "bt -r", although it does make
symbolic translations if possible. Perhaps that doesn't satisfy your
requirements?
For the moment, I'm able to get the task_struct * with following command :
ps | grep '>' | awk '{ print $5 };'
From that, the pointer of the thread_info would be available through
(task_struct *)->stack.
The "set" command, also available from "foreach", shows shows the
thread_info
pointer, although if you run it through a script you would have to strip the
brackets from the display.
But how to get all these together to finally ouptut a valid 'rd' command ?
Meaby another way to get this kernel stack dump would exist ?
Any help would be appreciate.
Regards,
Patrick Agrain
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
2:47 a.m.
Hello Dave,
Thanks for the tips. I'll try it.
I wrote a kernel module that crash the kernel after a timeout elapsing
by writing at 0xFFFFFFFF.
It seems that the 'bt -f' that I used to analyze the crash displays the
pt_regs structure provided by error_code() splitted over 2 consecutive
frames.
That's why I wanted to cross-check with a 'raw' dump.
Regards,
Patrick
Le 06/12/2013 14:57, Dave Anderson a écrit :
----- Original Message -----
> Hello,
>
> We wish to call 'crash' with a script file to automate some basic
> command and output the result on a text file.
>
> This works perfectly for basic commands such as bt, log, mod and so on,
> especially when no input data is required.
>
> We try now to dump the kernel stack through a 'rd <@> <size>'.
> My problem is to "extract" this <@> and to "input" it in
this 'rd' command.
You can dump the full kernel stack with "bt -r", although it does make
symbolic translations if possible. Perhaps that doesn't satisfy your
requirements?
> For the moment, I'm able to get the task_struct * with following command :
> ps | grep '>' | awk '{ print $5 };'
>
> From that, the pointer of the thread_info would be available through
> (task_struct *)->stack.
The "set" command, also available from "foreach", shows shows the
thread_info
pointer, although if you run it through a script you would have to strip the
brackets from the display.
> But how to get all these together to finally ouptut a valid 'rd' command ?
> Meaby another way to get this kernel stack dump would exist ?
>
> Any help would be appreciate.
> Regards,
> Patrick Agrain
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
2:56 a.m.
New subject: SOLVED - Retrieve pointer value for scripting
Hello,
In order to share the solution, here is the command that fulfill our
need : get a raw dump of the kernel stack of the offending task.
set -p
task -R stack | grep "stack =" |awk '{print $3}'|cut -f1 -d","
|awk
'{print "rd "$1 " " 0x800 }' > rd_kstack
< rd_kstack
PS : the 'bt- r' will be a good alternate... when we will get more
familiar with the 'crash' utility.
Again, thanks to all.
Regards,
Patrick
10:06 a.m.
New subject: About the use of 'gcore'
Hello all,
Now that we have dumped the kernel stack, I'm intesresting in the user
process from which we came just before the 'panic'.
Googling around, I found mention of the 'gcore' extension.
I compiled version 1.22 and installed it.
Using it on crash 6.1.0-1.el6, I get a file core.845.bash on process
'bash' (in which I trigger a kernel panic) :
crash> gcore -v 1 845
gcore: Opening file core.845.bash ...
gcore: done.
gcore: Writing ELF header ...
gcore: done.
gcore: Retrieving and writing note information ...
gcore: done.
gcore: Writing PT_NOTE program header ...
gcore: done.
gcore: Writing PT_LOAD program headers ...
gcore: done.
gcore: Writing PT_LOAD segment ...
gcore: PT_LOAD[0]: 8048000 - 8048000
gcore: PT_LOAD[1]: 80e2000 - 80e9000
gcore: PT_LOAD[2]: 80e9000 - 80ed000
gcore: PT_LOAD[3]: 94a2000 - 94d1000
gcore: PT_LOAD[4]: b7374000 - b7374000
gcore: PT_LOAD[5]: b7375000 - b7376000
gcore: PT_LOAD[6]: b7376000 - b7377000
gcore: PT_LOAD[7]: b7377000 - b7377000
gcore: PT_LOAD[8]: b737e000 - b737e000
gcore: PT_LOAD[9]: b737f000 - b737f000
gcore: PT_LOAD[10]: b73bb000 - b73bb000
gcore: PT_LOAD[11]: b75bb000 - b75bb000
gcore: PT_LOAD[12]: b75c7000 - b75c8000
gcore: PT_LOAD[13]: b75c8000 - b75c9000
gcore: PT_LOAD[14]: b75c9000 - b75ca000
gcore: PT_LOAD[15]: b75ca000 - b75ca000
gcore: PT_LOAD[16]: b7756000 - b7758000
gcore: PT_LOAD[17]: b7758000 - b7759000
gcore: PT_LOAD[18]: b7759000 - b775c000
gcore: PT_LOAD[19]: b775c000 - b775c000
gcore: PT_LOAD[20]: b775f000 - b7760000
gcore: PT_LOAD[21]: b7760000 - b7761000
gcore: PT_LOAD[22]: b7761000 - b7761000
gcore: PT_LOAD[23]: b7764000 - b7765000
gcore: PT_LOAD[24]: b7769000 - b776a000
gcore: PT_LOAD[25]: b776a000 - b776b000
gcore: PT_LOAD[26]: b776b000 - b776b000
gcore: PT_LOAD[27]: b7789000 - b778a000
gcore: PT_LOAD[28]: b778a000 - b778b000
gcore: PT_LOAD[29]: bfd07000 - bfd1d000
gcore: done.
Saved core.845.bash
crash>
So far, so good... But
Question: Are there anywhere some hints about how to use this core.<pid>
file ?
Thanks in advance.
Regards,
Patrick Agrain
10:52 a.m.
New subject: About the use of 'gcore'
----- Original Message -----
Hello all,
Now that we have dumped the kernel stack, I'm intesresting in the user
process from which we came just before the 'panic'.
Googling around, I found mention of the 'gcore' extension.
I compiled version 1.22 and installed it.
Using it on crash 6.1.0-1.el6, I get a file core.845.bash on process 'bash'
(in which I trigger a kernel panic) :
crash> gcore -v 1 845
gcore: Opening file core.845.bash ...
gcore: done.
gcore: Writing ELF header ...
gcore: done.
gcore: Retrieving and writing note information ...
gcore: done.
gcore: Writing PT_NOTE program header ...
gcore: done.
gcore: Writing PT_LOAD program headers ...
gcore: done.
gcore: Writing PT_LOAD segment ...
gcore: PT_LOAD[0]: 8048000 - 8048000
gcore: PT_LOAD[1]: 80e2000 - 80e9000
gcore: PT_LOAD[2]: 80e9000 - 80ed000
gcore: PT_LOAD[3]: 94a2000 - 94d1000
gcore: PT_LOAD[4]: b7374000 - b7374000
gcore: PT_LOAD[5]: b7375000 - b7376000
gcore: PT_LOAD[6]: b7376000 - b7377000
gcore: PT_LOAD[7]: b7377000 - b7377000
gcore: PT_LOAD[8]: b737e000 - b737e000
gcore: PT_LOAD[9]: b737f000 - b737f000
gcore: PT_LOAD[10]: b73bb000 - b73bb000
gcore: PT_LOAD[11]: b75bb000 - b75bb000
gcore: PT_LOAD[12]: b75c7000 - b75c8000
gcore: PT_LOAD[13]: b75c8000 - b75c9000
gcore: PT_LOAD[14]: b75c9000 - b75ca000
gcore: PT_LOAD[15]: b75ca000 - b75ca000
gcore: PT_LOAD[16]: b7756000 - b7758000
gcore: PT_LOAD[17]: b7758000 - b7759000
gcore: PT_LOAD[18]: b7759000 - b775c000
gcore: PT_LOAD[19]: b775c000 - b775c000
gcore: PT_LOAD[20]: b775f000 - b7760000
gcore: PT_LOAD[21]: b7760000 - b7761000
gcore: PT_LOAD[22]: b7761000 - b7761000
gcore: PT_LOAD[23]: b7764000 - b7765000
gcore: PT_LOAD[24]: b7769000 - b776a000
gcore: PT_LOAD[25]: b776a000 - b776b000
gcore: PT_LOAD[26]: b776b000 - b776b000
gcore: PT_LOAD[27]: b7789000 - b778a000
gcore: PT_LOAD[28]: b778a000 - b778b000
gcore: PT_LOAD[29]: bfd07000 - bfd1d000
gcore: done.
Saved core.845.bash
crash>
So far, so good... But
Question: Are there anywhere some hints about how to use this core.<pid> file
?
Thanks in advance.
Regards,
Patrick Agrain
It's meant to be used with gdb, similar to a normal core file generated if
the user-space process crashed and dropped a core file, or if it was created
by using the user-space /usr/bin/gcore command. So try this:
$ gdb /usr/bin/bash core.845.bash
Dave
12:12 p.m.
New subject: About the use of 'gcore'
Hi Patrick,
The following links may also be helpful to understand gdb and
it's usage for application core analysis.
http://web.eecs.umich.edu/~sugih/pointers/gdb_core.html
https://sourceware.org/gdb/onlinedocs/gdb/
-- BKS
On 17 December 2013 21:36, Patrick Agrain <patrick.agrain(a)alcatel-lucent.com
wrote:
Hello all,
Now that we have dumped the kernel stack, I'm intesresting in the user
process from which we came just before the 'panic'.
Googling around, I found mention of the 'gcore' extension.
I compiled version 1.22 and installed it.
Using it on crash 6.1.0-1.el6, I get a file core.845.bash on process
'bash' (in which I trigger a kernel panic) :
crash> gcore -v 1 845
gcore: Opening file core.845.bash ...
gcore: done.
gcore: Writing ELF header ...
gcore: done.
gcore: Retrieving and writing note information ...
gcore: done.
gcore: Writing PT_NOTE program header ...
gcore: done.
gcore: Writing PT_LOAD program headers ...
gcore: done.
gcore: Writing PT_LOAD segment ...
gcore: PT_LOAD[0]: 8048000 - 8048000
gcore: PT_LOAD[1]: 80e2000 - 80e9000
gcore: PT_LOAD[2]: 80e9000 - 80ed000
gcore: PT_LOAD[3]: 94a2000 - 94d1000
gcore: PT_LOAD[4]: b7374000 - b7374000
gcore: PT_LOAD[5]: b7375000 - b7376000
gcore: PT_LOAD[6]: b7376000 - b7377000
gcore: PT_LOAD[7]: b7377000 - b7377000
gcore: PT_LOAD[8]: b737e000 - b737e000
gcore: PT_LOAD[9]: b737f000 - b737f000
gcore: PT_LOAD[10]: b73bb000 - b73bb000
gcore: PT_LOAD[11]: b75bb000 - b75bb000
gcore: PT_LOAD[12]: b75c7000 - b75c8000
gcore: PT_LOAD[13]: b75c8000 - b75c9000
gcore: PT_LOAD[14]: b75c9000 - b75ca000
gcore: PT_LOAD[15]: b75ca000 - b75ca000
gcore: PT_LOAD[16]: b7756000 - b7758000
gcore: PT_LOAD[17]: b7758000 - b7759000
gcore: PT_LOAD[18]: b7759000 - b775c000
gcore: PT_LOAD[19]: b775c000 - b775c000
gcore: PT_LOAD[20]: b775f000 - b7760000
gcore: PT_LOAD[21]: b7760000 - b7761000
gcore: PT_LOAD[22]: b7761000 - b7761000
gcore: PT_LOAD[23]: b7764000 - b7765000
gcore: PT_LOAD[24]: b7769000 - b776a000
gcore: PT_LOAD[25]: b776a000 - b776b000
gcore: PT_LOAD[26]: b776b000 - b776b000
gcore: PT_LOAD[27]: b7789000 - b778a000
gcore: PT_LOAD[28]: b778a000 - b778b000
gcore: PT_LOAD[29]: bfd07000 - bfd1d000
gcore: done.
Saved core.845.bash
crash>
So far, so good... But
Question: Are there anywhere some hints about how to use this core.<pid>
file ?
Thanks in advance.
Regards,
Patrick Agrain
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
BKS
3:15 a.m.
New subject: About the use of 'gcore'
Hello all,
First of all, I wish a Happy New Year (with less crash, but still
enhanced tools...)
Thanks for the links, they were very useful.
I dig further in the way of analyzing the User Space, but it seems that
I'm linked to a dead-end way.
Below is a snapshot of kernel / userland stack dump.
What I've done :
- Crash is triggered by a page fault inside a kernel module (write 0 in
0xFFFFFFFF, classic).
- Using gcore to create the 'core.<pid>.bash (which is the user task
running at time of crash).
- Evaluating an EBP (between { }) chaining value (hypothesis), EIP value
(between [ ]) is then just pushed beside
The purpose of this study is to find a method to analyze futur crashes
from kernel space down to user space applications.
Do you have an idea about the cause of this non-dumping of the memory in
user-space ?
Should I use other extension as 'gcore' ?
Thank in advance.
Best regards,
Patrick Agrain
-------
===============================================================================
--------------------- Go down into User Space Territory
-----------------------
Last pt_regs of kernel stack is:
| pt_regs
00000001 094a5408 00000003 ..~......TJ..... |
bx cx dx
c2699fc0: 00000003 094a5408 bfd1b704 00000004 .....TJ......... | si di
bp ax
c2699fd0: 0000007b ffff007b c07e0000 00000033 {...{.....~.3... | ds es
fs gs
c2699fe0: 00000004 b776a416 00000073 00000246 ......v.s...F... |
orig_eax ip cs flags
c2699ff0: bfd1b6d8 0000007b | sp ss
v cccccccc cccccccc ....{........... | padding
|
|----------------------------------------------------------------|
|
(gdb) x/32xw 0xbfd1b680 |
0xbfd1b680: 0xbfd1b6d0 0x0000000f 0x094b4568 0x080c90b9 |
0xbfd1b690: 0x094b4568 0x080cd160 0x00001936 0x00000001 |
0xbfd1b6a0: 0x094ab9c8 0x00000000 0x094b4b48 0xbfd1b7c8 |
0xbfd1b6b0: 0x080ce9e8 0x094b4b48 0x094b4b48 0xbfd1b728 |
0xbfd1b6c0: 0x094aed28 0x00000020 0x00000000 0x00000070 |
0xbfd1b6d0: 0x094b4588 0x080cc080 |
0xb7698b43 <--|
0xb7757ff4
0xbfd1b6e0: 0xb76343b4 0x00000001 0x094a5408 0x00000003
0xbfd1b6f0: 0xb77584e0 0x080cc080 0xbfd1b728 0xb77584e0
|------------------------------------------ Hypothesis : this is an EBP
value...
v
0xbfd1b700: 0x00000003 {0xbfd1b72c} [0xb7635c90] 0xb77584e0
0xbfd1b710: 0x094a5408 0x00000003 0x094b4b48 0xbfd1b7c8
0xbfd1b720: 0xb7757ff4 0xb77584e0 0x0000000a {0xbfd1b750}
0xbfd1b730: [0xb7634e80] 0xb77584e0 0x094a5408 0x00000003
0xbfd1b740: 0x0000000a 0xb7757ff4 0xb77584e0 0x0000000a
0xbfd1b750: {0xbfd1b768} [0xb7637d2a] 0xb77584e0 0x0000000a
0xbfd1b760: 0xb7757ff4 0xb77584e0 {0xbfd1b788} [0xb76312b5]>-|
0xbfd1b770: 0xb77584e0 0x0000000a 0xb75c9940 0x094a3e48 |
0xbfd1b780: 0x00000001 0x00000000 0x00000000 0x0809b64b |
|
Disassemble Try: EIP@0xb76312b5
<---------------------------------------------|
(gdb) disassemble 0xb7631200, 0xb7631300
Dump of assembler code from 0xb7631200 to 0xb7631300:
0xb7631200: Cannot access memory at address 0xb7631200
(gdb)
----------
Le 17/12/2013 19:12, Buland Kumar Singh a écrit :
Hi Patrick,
The following links may also be helpful to understand gdb and
it's usage for application core analysis.
http://web.eecs.umich.edu/~sugih/pointers/gdb_core.html
<http://web.eecs.umich.edu/%7Esugih/pointers/gdb_core.html>
https://sourceware.org/gdb/onlinedocs/gdb/
-- BKS
On 17 December 2013 21:36, Patrick Agrain
<patrick.agrain(a)alcatel-lucent.com
<mailto:patrick.agrain@alcatel-lucent.com>> wrote:
Hello all,
Now that we have dumped the kernel stack, I'm intesresting in the
user process from which we came just before the 'panic'.
Googling around, I found mention of the 'gcore' extension.
I compiled version 1.22 and installed it.
Using it on crash 6.1.0-1.el6, I get a file core.845.bash on
process 'bash' (in which I trigger a kernel panic) :
crash> gcore -v 1 845
gcore: Opening file core.845.bash ...
gcore: done.
gcore: Writing ELF header ...
gcore: done.
gcore: Retrieving and writing note information ...
gcore: done.
gcore: Writing PT_NOTE program header ...
gcore: done.
gcore: Writing PT_LOAD program headers ...
gcore: done.
gcore: Writing PT_LOAD segment ...
gcore: PT_LOAD[0]: 8048000 - 8048000
gcore: PT_LOAD[1]: 80e2000 - 80e9000
gcore: PT_LOAD[2]: 80e9000 - 80ed000
gcore: PT_LOAD[3]: 94a2000 - 94d1000
gcore: PT_LOAD[4]: b7374000 - b7374000
gcore: PT_LOAD[5]: b7375000 - b7376000
gcore: PT_LOAD[6]: b7376000 - b7377000
gcore: PT_LOAD[7]: b7377000 - b7377000
gcore: PT_LOAD[8]: b737e000 - b737e000
gcore: PT_LOAD[9]: b737f000 - b737f000
gcore: PT_LOAD[10]: b73bb000 - b73bb000
gcore: PT_LOAD[11]: b75bb000 - b75bb000
gcore: PT_LOAD[12]: b75c7000 - b75c8000
gcore: PT_LOAD[13]: b75c8000 - b75c9000
gcore: PT_LOAD[14]: b75c9000 - b75ca000
gcore: PT_LOAD[15]: b75ca000 - b75ca000
gcore: PT_LOAD[16]: b7756000 - b7758000
gcore: PT_LOAD[17]: b7758000 - b7759000
gcore: PT_LOAD[18]: b7759000 - b775c000
gcore: PT_LOAD[19]: b775c000 - b775c000
gcore: PT_LOAD[20]: b775f000 - b7760000
gcore: PT_LOAD[21]: b7760000 - b7761000
gcore: PT_LOAD[22]: b7761000 - b7761000
gcore: PT_LOAD[23]: b7764000 - b7765000
gcore: PT_LOAD[24]: b7769000 - b776a000
gcore: PT_LOAD[25]: b776a000 - b776b000
gcore: PT_LOAD[26]: b776b000 - b776b000
gcore: PT_LOAD[27]: b7789000 - b778a000
gcore: PT_LOAD[28]: b778a000 - b778b000
gcore: PT_LOAD[29]: bfd07000 - bfd1d000
gcore: done.
Saved core.845.bash
crash>
So far, so good... But
Question: Are there anywhere some hints about how to use this
core.<pid> file ?
Thanks in advance.
Regards,
Patrick Agrain
--
Crash-utility mailing list
Crash-utility(a)redhat.com <mailto:Crash-utility@redhat.com>
https://www.redhat.com/mailman/listinfo/crash-utility
--
BKS
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
3:08 p.m.
New subject: About the use of 'gcore'
----- Original Message -----
Hello all,
First of all, I wish a Happy New Year (with less crash, but still enhanced
tools...)
Thanks for the links, they were very useful.
I dig further in the way of analyzing the User Space, but it seems that I'm
linked to a dead-end way.
Below is a snapshot of kernel / userland stack dump.
What I've done :
- Crash is triggered by a page fault inside a kernel module (write 0 in
0xFFFFFFFF, classic).
- Using gcore to create the 'core.<pid>.bash (which is the user task running
at time of crash).
I'm curious as to how the bash task was related to the module crash?
Did the bash task write to a procfs interface that the module created
to then generate the "write 0 to 0xFFFFFFFF"? Does the crash utility
indicate that the bash task is the panic task? And if so, what does
its "bt" show? (i.e., the kernel-mode backtrace)
- Evaluating an EBP (between { }) chaining value (hypothesis), EIP
value
(between [ ]) is then just pushed beside
The purpose of this study is to find a method to analyze futur crashes from
kernel space down to user space applications.
Do you have an idea about the cause of this non-dumping of the memory in
user-space ?
Should I use other extension as 'gcore' ?
Thank in advance.
Best regards,
Patrick Agrain
-------
===============================================================================
--------------------- Go down into User Space Territory
-----------------------
Last pt_regs of kernel stack is:
| pt_regs
00000001 094a5408 00000003 ..~......TJ..... | bx cx dx
c2699fc0: 00000003 094a5408 bfd1b704 00000004 .....TJ......... | si di bp ax
c2699fd0: 0000007b ffff007b c07e0000 00000033 {...{.....~.3... | ds es fs gs
c2699fe0: 00000004 b776a416 00000073 00000246 ......v.s...F... | orig_eax ip
cs flags
c2699ff0: bfd1b6d8 0000007b | sp ss
v cccccccc cccccccc ....{........... | padding
|
|----------------------------------------------------------------|
|
(gdb) x/32xw 0xbfd1b680 |
0xbfd1b680: 0xbfd1b6d0 0x0000000f 0x094b4568 0x080c90b9 |
0xbfd1b690: 0x094b4568 0x080cd160 0x00001936 0x00000001 |
0xbfd1b6a0: 0x094ab9c8 0x00000000 0x094b4b48 0xbfd1b7c8 |
0xbfd1b6b0: 0x080ce9e8 0x094b4b48 0x094b4b48 0xbfd1b728 |
0xbfd1b6c0: 0x094aed28 0x00000020 0x00000000 0x00000070 |
0xbfd1b6d0: 0x094b4588 0x080cc080 |
0xb7698b43 <--|
0xb7757ff4
0xbfd1b6e0: 0xb76343b4 0x00000001 0x094a5408 0x00000003
0xbfd1b6f0: 0xb77584e0 0x080cc080 0xbfd1b728 0xb77584e0
|------------------------------------------ Hypothesis : this is an EBP
|value...
v
0xbfd1b700: 0x00000003 {0xbfd1b72c} [0xb7635c90] 0xb77584e0
0xbfd1b710: 0x094a5408 0x00000003 0x094b4b48 0xbfd1b7c8
0xbfd1b720: 0xb7757ff4 0xb77584e0 0x0000000a {0xbfd1b750}
0xbfd1b730: [0xb7634e80] 0xb77584e0 0x094a5408 0x00000003
0xbfd1b740: 0x0000000a 0xb7757ff4 0xb77584e0 0x0000000a
0xbfd1b750: {0xbfd1b768} [0xb7637d2a] 0xb77584e0 0x0000000a
0xbfd1b760: 0xb7757ff4 0xb77584e0 {0xbfd1b788} [0xb76312b5] >-|
0xbfd1b770: 0xb77584e0 0x0000000a 0xb75c9940 0x094a3e48 |
0xbfd1b780: 0x00000001 0x00000000 0x00000000 0x0809b64b |
|
Disassemble Try: EIP@0xb76312b5
<---------------------------------------------|
(gdb) disassemble 0xb7631200, 0xb7631300
Dump of assembler code from 0xb7631200 to 0xb7631300:
0xb7631200: Cannot access memory at address 0xb7631200
(gdb)
----------
Anyway, I'm guessing that the 0xb76312b5 IP address is in some
library, probably libc? If you do a "vm" on the active bash task
from within the crash utility, you will see where it comes from.
Try reading the user-space address from the crash utility to see
if it was available to copy to the core.<pid>.bash file, i.e.,
try this command:
crash> rd -u 0xb76312b5
The command above presumes that you are in the context of the
"bash" task while running crash. (i.e., if you enter "set" alone,
it shows that particular task)
Dave
Le 17/12/2013 19:12, Buland Kumar Singh a écrit :
Hi Patrick,
The following links may also be helpful to understand gdb and
it's usage for application core analysis.
http://web.eecs.umich.edu/~sugih/pointers/gdb_core.html
https://sourceware.org/gdb/onlinedocs/gdb/
-- BKS
On 17 December 2013 21:36, Patrick Agrain < patrick.agrain(a)alcatel-lucent.com
> wrote:
Hello all,
Now that we have dumped the kernel stack, I'm intesresting in the user
process from which we came just before the 'panic'.
Googling around, I found mention of the 'gcore' extension.
I compiled version 1.22 and installed it.
Using it on crash 6.1.0-1.el6, I get a file core.845.bash on process 'bash'
(in which I trigger a kernel panic) :
crash> gcore -v 1 845
gcore: Opening file core.845.bash ...
gcore: done.
gcore: Writing ELF header ...
gcore: done.
gcore: Retrieving and writing note information ...
gcore: done.
gcore: Writing PT_NOTE program header ...
gcore: done.
gcore: Writing PT_LOAD program headers ...
gcore: done.
gcore: Writing PT_LOAD segment ...
gcore: PT_LOAD[0]: 8048000 - 8048000
gcore: PT_LOAD[1]: 80e2000 - 80e9000
gcore: PT_LOAD[2]: 80e9000 - 80ed000
gcore: PT_LOAD[3]: 94a2000 - 94d1000
gcore: PT_LOAD[4]: b7374000 - b7374000
gcore: PT_LOAD[5]: b7375000 - b7376000
gcore: PT_LOAD[6]: b7376000 - b7377000
gcore: PT_LOAD[7]: b7377000 - b7377000
gcore: PT_LOAD[8]: b737e000 - b737e000
gcore: PT_LOAD[9]: b737f000 - b737f000
gcore: PT_LOAD[10]: b73bb000 - b73bb000
gcore: PT_LOAD[11]: b75bb000 - b75bb000
gcore: PT_LOAD[12]: b75c7000 - b75c8000
gcore: PT_LOAD[13]: b75c8000 - b75c9000
gcore: PT_LOAD[14]: b75c9000 - b75ca000
gcore: PT_LOAD[15]: b75ca000 - b75ca000
gcore: PT_LOAD[16]: b7756000 - b7758000
gcore: PT_LOAD[17]: b7758000 - b7759000
gcore: PT_LOAD[18]: b7759000 - b775c000
gcore: PT_LOAD[19]: b775c000 - b775c000
gcore: PT_LOAD[20]: b775f000 - b7760000
gcore: PT_LOAD[21]: b7760000 - b7761000
gcore: PT_LOAD[22]: b7761000 - b7761000
gcore: PT_LOAD[23]: b7764000 - b7765000
gcore: PT_LOAD[24]: b7769000 - b776a000
gcore: PT_LOAD[25]: b776a000 - b776b000
gcore: PT_LOAD[26]: b776b000 - b776b000
gcore: PT_LOAD[27]: b7789000 - b778a000
gcore: PT_LOAD[28]: b778a000 - b778b000
gcore: PT_LOAD[29]: bfd07000 - bfd1d000
gcore: done.
Saved core.845.bash
crash>
So far, so good... But
Question: Are there anywhere some hints about how to use this core.<pid> file
?
Thanks in advance.
Regards,
Patrick Agrain
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
BKS
--
Crash-utility mailing list Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
12:47 a.m.
New subject: About the use of 'gcore'
Hello Dave,
Thanks for the answers. I'll check your suggestions.
More answers in the text...
Le 06/01/2014 22:08, Dave Anderson a écrit :
----- Original Message -----
> Hello all,
>
> First of all, I wish a Happy New Year (with less crash, but still enhanced
> tools...)
>
> Thanks for the links, they were very useful.
> I dig further in the way of analyzing the User Space, but it seems that I'm
> linked to a dead-end way.
> Below is a snapshot of kernel / userland stack dump.
>
> What I've done :
> - Crash is triggered by a page fault inside a kernel module (write 0 in
> 0xFFFFFFFF, classic).
> - Using gcore to create the 'core.<pid>.bash (which is the user task
running
> at time of crash).
I'm curious as to how the bash task was related to the module crash?
Did the bash task write to a procfs interface that the module created
to then generate the "write 0 to 0xFFFFFFFF"? Does the crash utility
indicate that the bash task is the panic task? And if so, what does
its "bt" show? (i.e., the kernel-mode backtrace)
That's correct.
I wrote a kernel module (timecrash.ko) to do the page fault after a
timeout elapsed.
The timer is triggered by: echo <timeout_in_second> > /proc/tocrashme
The 'bt' command shows following:
PID: 892 TASK: c274e550 CPU: 0 COMMAND: "bash"
#0 [c2699d20] crash_kexec at c0492ecc
#1 [c2699d78] oops_end at c07ebbb2
#2 [c2699d90] no_context at c042d389
#3 [c2699db8] __bad_area_nosemaphore at c042d4b3
#4 [c2699df8] bad_area_nosemaphore at c042d57d
#5 [c2699e04] __do_page_fault at c042da5c
#6 [c2699e88] do_page_fault at c07ed531
#7 [c2699ea4] error_code (via page_fault) at c07eaf3d
EAX: 00000028 EBX: 00000003 ECX: c09e6514 EDX: 00000000 EBP:
c2699f20
DS: 007b ESI: 00000000 ES: 007b EDI: 094a5408 GS: 00e0
CS: 0060 EIP: f87ad1c6 ERR: ffffffff EFLAGS: 00010296
#8 [c2699ed8] proc_crash_setdelay at f87ad1c6 [timecrash]
#9 [c2699f24] proc_file_write at c0572856
#10 [c2699f44] proc_reg_write at c056d5dd
#11 [c2699f68] vfs_write at c051f637
#12 [c2699f90] sys_write at c051ff38
#13 [c2699fb0] system_call at c07ea7ad
EAX: 00000004 EBX: 00000001 ECX: 094a5408 EDX: 00000003
DS: 007b ESI: 00000003 ES: 007b EDI: 094a5408
SS: 007b ESP: bfd1b6d8 EBP: bfd1b704 GS: 0033
CS: 0073 EIP: b776a416 ERR: 00000004 EFLAGS: 00000246
> - Evaluating an EBP (between { }) chaining value (hypothesis), EIP value
> (between [ ]) is then just pushed beside
>
> The purpose of this study is to find a method to analyze futur crashes from
> kernel space down to user space applications.
>
> Do you have an idea about the cause of this non-dumping of the memory in
> user-space ?
> Should I use other extension as 'gcore' ?
>
> Thank in advance.
> Best regards,
> Patrick Agrain
>
>
> -------
> ===============================================================================
> --------------------- Go down into User Space Territory
> -----------------------
>
> Last pt_regs of kernel stack is:
> | pt_regs
> 00000001 094a5408 00000003 ..~......TJ..... | bx cx dx
> c2699fc0: 00000003 094a5408 bfd1b704 00000004 .....TJ......... | si di bp ax
> c2699fd0: 0000007b ffff007b c07e0000 00000033 {...{.....~.3... | ds es fs gs
> c2699fe0: 00000004 b776a416 00000073 00000246 ......v.s...F... | orig_eax ip
> cs flags
> c2699ff0: bfd1b6d8 0000007b | sp ss
> v cccccccc cccccccc ....{........... | padding
> |
> |----------------------------------------------------------------|
> |
> (gdb) x/32xw 0xbfd1b680 |
> 0xbfd1b680: 0xbfd1b6d0 0x0000000f 0x094b4568 0x080c90b9 |
> 0xbfd1b690: 0x094b4568 0x080cd160 0x00001936 0x00000001 |
> 0xbfd1b6a0: 0x094ab9c8 0x00000000 0x094b4b48 0xbfd1b7c8 |
> 0xbfd1b6b0: 0x080ce9e8 0x094b4b48 0x094b4b48 0xbfd1b728 |
> 0xbfd1b6c0: 0x094aed28 0x00000020 0x00000000 0x00000070 |
> 0xbfd1b6d0: 0x094b4588 0x080cc080 |
> 0xb7698b43 <--|
> 0xb7757ff4
> 0xbfd1b6e0: 0xb76343b4 0x00000001 0x094a5408 0x00000003
> 0xbfd1b6f0: 0xb77584e0 0x080cc080 0xbfd1b728 0xb77584e0
>
> |------------------------------------------ Hypothesis : this is an EBP
> |value...
> v
> 0xbfd1b700: 0x00000003 {0xbfd1b72c} [0xb7635c90] 0xb77584e0
> 0xbfd1b710: 0x094a5408 0x00000003 0x094b4b48 0xbfd1b7c8
> 0xbfd1b720: 0xb7757ff4 0xb77584e0 0x0000000a {0xbfd1b750}
> 0xbfd1b730: [0xb7634e80] 0xb77584e0 0x094a5408 0x00000003
> 0xbfd1b740: 0x0000000a 0xb7757ff4 0xb77584e0 0x0000000a
> 0xbfd1b750: {0xbfd1b768} [0xb7637d2a] 0xb77584e0 0x0000000a
> 0xbfd1b760: 0xb7757ff4 0xb77584e0 {0xbfd1b788} [0xb76312b5] >-|
> 0xbfd1b770: 0xb77584e0 0x0000000a 0xb75c9940 0x094a3e48 |
> 0xbfd1b780: 0x00000001 0x00000000 0x00000000 0x0809b64b |
> |
> Disassemble Try: EIP@0xb76312b5
> <---------------------------------------------|
> (gdb) disassemble 0xb7631200, 0xb7631300
> Dump of assembler code from 0xb7631200 to 0xb7631300:
> 0xb7631200: Cannot access memory at address 0xb7631200
> (gdb)
> ----------
Anyway, I'm guessing that the 0xb76312b5 IP address is in some
library, probably libc? If you do a "vm" on the active bash task
from within the crash utility, you will see where it comes from.
Try reading the user-space address from the crash utility to see
if it was available to copy to the core.<pid>.bash file, i.e.,
try this command:
crash> rd -u 0xb76312b5
The command above presumes that you are in the context of the
"bash" task while running crash. (i.e., if you enter "set" alone,
it shows that particular task)
Dave
> Le 17/12/2013 19:12, Buland Kumar Singh a écrit :
>
>
>
> Hi Patrick,
>
> The following links may also be helpful to understand gdb and
> it's usage for application core analysis.
>
> http://web.eecs.umich.edu/~sugih/pointers/gdb_core.html
> https://sourceware.org/gdb/onlinedocs/gdb/
>
> -- BKS
>
>
> On 17 December 2013 21:36, Patrick Agrain < patrick.agrain(a)alcatel-lucent.com
>> wrote:
>
>
> Hello all,
>
> Now that we have dumped the kernel stack, I'm intesresting in the user
> process from which we came just before the 'panic'.
> Googling around, I found mention of the 'gcore' extension.
>
> I compiled version 1.22 and installed it.
> Using it on crash 6.1.0-1.el6, I get a file core.845.bash on process 'bash'
> (in which I trigger a kernel panic) :
>
>
>
> crash> gcore -v 1 845
> gcore: Opening file core.845.bash ...
> gcore: done.
> gcore: Writing ELF header ...
> gcore: done.
> gcore: Retrieving and writing note information ...
> gcore: done.
> gcore: Writing PT_NOTE program header ...
> gcore: done.
> gcore: Writing PT_LOAD program headers ...
> gcore: done.
> gcore: Writing PT_LOAD segment ...
> gcore: PT_LOAD[0]: 8048000 - 8048000
> gcore: PT_LOAD[1]: 80e2000 - 80e9000
> gcore: PT_LOAD[2]: 80e9000 - 80ed000
> gcore: PT_LOAD[3]: 94a2000 - 94d1000
> gcore: PT_LOAD[4]: b7374000 - b7374000
> gcore: PT_LOAD[5]: b7375000 - b7376000
> gcore: PT_LOAD[6]: b7376000 - b7377000
> gcore: PT_LOAD[7]: b7377000 - b7377000
> gcore: PT_LOAD[8]: b737e000 - b737e000
> gcore: PT_LOAD[9]: b737f000 - b737f000
> gcore: PT_LOAD[10]: b73bb000 - b73bb000
> gcore: PT_LOAD[11]: b75bb000 - b75bb000
> gcore: PT_LOAD[12]: b75c7000 - b75c8000
> gcore: PT_LOAD[13]: b75c8000 - b75c9000
> gcore: PT_LOAD[14]: b75c9000 - b75ca000
> gcore: PT_LOAD[15]: b75ca000 - b75ca000
> gcore: PT_LOAD[16]: b7756000 - b7758000
> gcore: PT_LOAD[17]: b7758000 - b7759000
> gcore: PT_LOAD[18]: b7759000 - b775c000
> gcore: PT_LOAD[19]: b775c000 - b775c000
> gcore: PT_LOAD[20]: b775f000 - b7760000
> gcore: PT_LOAD[21]: b7760000 - b7761000
> gcore: PT_LOAD[22]: b7761000 - b7761000
> gcore: PT_LOAD[23]: b7764000 - b7765000
> gcore: PT_LOAD[24]: b7769000 - b776a000
> gcore: PT_LOAD[25]: b776a000 - b776b000
> gcore: PT_LOAD[26]: b776b000 - b776b000
> gcore: PT_LOAD[27]: b7789000 - b778a000
> gcore: PT_LOAD[28]: b778a000 - b778b000
> gcore: PT_LOAD[29]: bfd07000 - bfd1d000
> gcore: done.
> Saved core.845.bash
> crash>
>
> So far, so good... But
>
> Question: Are there anywhere some hints about how to use this core.<pid> file
> ?
>
> Thanks in advance.
> Regards,
> Patrick Agrain
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
>
>
> --
> BKS
>
>
> --
> Crash-utility mailing list Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
7:49 a.m.
New subject: About the use of 'gcore'
----- Original Message -----
Hello Dave,
Thanks for the answers. I'll check your suggestions.
More answers in the text...
...
> I'm curious as to how the bash task was related to the module crash?
> Did the bash task write to a procfs interface that the module created
> to then generate the "write 0 to 0xFFFFFFFF"? Does the crash utility
> indicate that the bash task is the panic task? And if so, what does
> its "bt" show? (i.e., the kernel-mode backtrace)
That's correct.
I wrote a kernel module (timecrash.ko) to do the page fault after a timeout elapsed.
The timer is triggered by: echo <timeout_in_second> > /proc/tocrashme
The 'bt' command shows following:
PID: 892 TASK: c274e550 CPU: 0 COMMAND: "bash"
#0 [c2699d20] crash_kexec at c0492ecc
#1 [c2699d78] oops_end at c07ebbb2
#2 [c2699d90] no_context at c042d389
#3 [c2699db8] __bad_area_nosemaphore at c042d4b3
#4 [c2699df8] bad_area_nosemaphore at c042d57d
#5 [c2699e04] __do_page_fault at c042da5c
#6 [c2699e88] do_page_fault at c07ed531
#7 [c2699ea4] error_code (via page_fault) at c07eaf3d
EAX: 00000028 EBX: 00000003 ECX: c09e6514 EDX: 00000000 EBP: c2699f20
DS: 007b ESI: 00000000 ES: 007b EDI: 094a5408 GS: 00e0
CS: 0060 EIP: f87ad1c6 ERR: ffffffff EFLAGS: 00010296
#8 [c2699ed8] proc_crash_setdelay at f87ad1c6 [timecrash]
#9 [c2699f24] proc_file_write at c0572856
#10 [c2699f44] proc_reg_write at c056d5dd
#11 [c2699f68] vfs_write at c051f637
#12 [c2699f90] sys_write at c051ff38
#13 [c2699fb0] system_call at c07ea7ad
EAX: 00000004 EBX: 00000001 ECX: 094a5408 EDX: 00000003
DS: 007b ESI: 00000003 ES: 007b EDI: 094a5408
SS: 007b ESP: bfd1b6d8 EBP: bfd1b704 GS: 0033
CS: 0073 EIP: b776a416 ERR: 00000004 EFLAGS: 00000246
OK, that's good, but you didn't answer the query below:
>> |
>> Disassemble Try: EIP@0xb76312b5
>> <---------------------------------------------|
>> (gdb) disassemble 0xb7631200, 0xb7631300
>> Dump of assembler code from 0xb7631200 to 0xb7631300:
>> 0xb7631200: Cannot access memory at address 0xb7631200
>> (gdb)
>> ----------
> Anyway, I'm guessing that the 0xb76312b5 IP address is in some
> library, probably libc? If you do a "vm" on the active bash task
> from within the crash utility, you will see where it comes from.
> Try reading the user-space address from the crash utility to see
> if it was available to copy to the core.<pid>.bash file, i.e.,
> try this command:
>
> crash> rd -u 0xb76312b5
>
> The command above presumes that you are in the context of the
> "bash" task while running crash. (i.e., if you enter "set"
alone,
> it shows that particular task)
>
> Dave
1:39 a.m.
New subject: About the use of 'gcore'
Hello everybody,
After complete re-installation of the machine, I'm back on stage.
Dave, to answer to your question, on gdb, the "bt" shows following:
(gdb) bt
#0 0xb776a416 in __kernel_vsyscall ()
#1 0xb7698b43 in __write_nocancel () from /lib/libc.so.6
#2 0xb76343b4 in _IO_new_file_write () from /lib/libc.so.6
#3 0xb7635c90 in _IO_new_do_write () from /lib/libc.so.6
#4 0xb7634e80 in _IO_new_file_overflow () from /lib/libc.so.6
#5 0xb7637d2a in __overflow () from /lib/libc.so.6
#6 0xb76312b5 in putc () from /lib/libc.so.6
#7 0x0809b64b in echo_builtin ()
So your guess was correct, most of the functions are from libc. Also
stated with the 'vm' command.
Performing again the whole process, I think that I now know where I was
wrong.
The error message at the origin of this thread was due to the fact that
I do not use the correct "bash" binary when launching 'gdb'. I made a
mismatch between the "bash" on the crashed system and the "bash" on
the
machine where I analyze the crash.
Using the correct "bash" allows to dump the memory and disassemble the
code without problem.
BTW, using the 'vm' and 'vtop' command (Thanks Dave for the tip) offers
interesting information that I will explore later.
For now, I wrote a user-space program that will trigger the page fault,
so that I can more easily make the relationship between assembly and C
language.
To be followed.
Patrick Agrain
Le 06/01/2014 22:08, Dave Anderson a écrit :
as evidenced by the fact that you were able to at least read stack address
bfd1b6d8 from gdb. But it's not clear -- after you walked through the
user stack -- that the dumpfile contains the text page that you're trying
to disassemble. I don't know why it wouldn't be available given that
it's a "hot" code path, and shouldn't be swapped out or otherwise not
mapped.
I'm also presuming that your user-stack walk-through is correct.
When you enter gdb, what does the "bt" command show?
Dave
----- Original Message -----
> Hello all,
>
> First of all, I wish a Happy New Year (with less crash, but still enhanced
> tools...)
>
> Thanks for the links, they were very useful.
> I dig further in the way of analyzing the User Space, but it seems that I'm
> linked to a dead-end way.
> Below is a snapshot of kernel / userland stack dump.
>
> What I've done :
> - Crash is triggered by a page fault inside a kernel module (write 0 in
> 0xFFFFFFFF, classic).
> - Using gcore to create the 'core.<pid>.bash (which is the user task
running
> at time of crash).
I'm curious as to how the bash task was related to the module crash?
Did the bash task write to a procfs interface that the module created
to then generate the "write 0 to 0xFFFFFFFF"? Does the crash utility
indicate that the bash task is the panic task? And if so, what does
its "bt" show? (i.e., the kernel-mode backtrace)
> - Evaluating an EBP (between { }) chaining value (hypothesis), EIP value
> (between [ ]) is then just pushed beside
>
> The purpose of this study is to find a method to analyze futur crashes from
> kernel space down to user space applications.
>
> Do you have an idea about the cause of this non-dumping of the memory in
> user-space ?
> Should I use other extension as 'gcore' ?
>
> Thank in advance.
> Best regards,
> Patrick Agrain
>
>
> -------
> ===============================================================================
> --------------------- Go down into User Space Territory
> -----------------------
>
> Last pt_regs of kernel stack is:
> | pt_regs
> 00000001 094a5408 00000003 ..~......TJ..... | bx cx dx
> c2699fc0: 00000003 094a5408 bfd1b704 00000004 .....TJ......... | si di bp ax
> c2699fd0: 0000007b ffff007b c07e0000 00000033 {...{.....~.3... | ds es fs gs
> c2699fe0: 00000004 b776a416 00000073 00000246 ......v.s...F... | orig_eax ip
> cs flags
> c2699ff0: bfd1b6d8 0000007b | sp ss
> v cccccccc cccccccc ....{........... | padding
> |
> |----------------------------------------------------------------|
> |
> (gdb) x/32xw 0xbfd1b680 |
> 0xbfd1b680: 0xbfd1b6d0 0x0000000f 0x094b4568 0x080c90b9 |
> 0xbfd1b690: 0x094b4568 0x080cd160 0x00001936 0x00000001 |
> 0xbfd1b6a0: 0x094ab9c8 0x00000000 0x094b4b48 0xbfd1b7c8 |
> 0xbfd1b6b0: 0x080ce9e8 0x094b4b48 0x094b4b48 0xbfd1b728 |
> 0xbfd1b6c0: 0x094aed28 0x00000020 0x00000000 0x00000070 |
> 0xbfd1b6d0: 0x094b4588 0x080cc080 |
> 0xb7698b43 <--|
> 0xb7757ff4
> 0xbfd1b6e0: 0xb76343b4 0x00000001 0x094a5408 0x00000003
> 0xbfd1b6f0: 0xb77584e0 0x080cc080 0xbfd1b728 0xb77584e0
>
> |------------------------------------------ Hypothesis : this is an EBP
> |value...
> v
> 0xbfd1b700: 0x00000003 {0xbfd1b72c} [0xb7635c90] 0xb77584e0
> 0xbfd1b710: 0x094a5408 0x00000003 0x094b4b48 0xbfd1b7c8
> 0xbfd1b720: 0xb7757ff4 0xb77584e0 0x0000000a {0xbfd1b750}
> 0xbfd1b730: [0xb7634e80] 0xb77584e0 0x094a5408 0x00000003
> 0xbfd1b740: 0x0000000a 0xb7757ff4 0xb77584e0 0x0000000a
> 0xbfd1b750: {0xbfd1b768} [0xb7637d2a] 0xb77584e0 0x0000000a
> 0xbfd1b760: 0xb7757ff4 0xb77584e0 {0xbfd1b788} [0xb76312b5] >-|
> 0xbfd1b770: 0xb77584e0 0x0000000a 0xb75c9940 0x094a3e48 |
> 0xbfd1b780: 0x00000001 0x00000000 0x00000000 0x0809b64b |
> |
> Disassemble Try: EIP@0xb76312b5
> <---------------------------------------------|
> (gdb) disassemble 0xb7631200, 0xb7631300
> Dump of assembler code from 0xb7631200 to 0xb7631300:
> 0xb7631200: Cannot access memory at address 0xb7631200
> (gdb)
> ----------
Anyway, I'm guessing that the 0xb76312b5 IP address is in some
library, probably libc? If you do a "vm" on the active bash task
from within the crash utility, you will see where it comes from.
Try reading the user-space address from the crash utility to see
if it was available to copy to the core.<pid>.bash file, i.e.,
try this command:
crash> rd -u 0xb76312b5
The command above presumes that you are in the context of the
"bash" task while running crash. (i.e., if you enter "set" alone,
it shows that particular task)
Dave
> Le 17/12/2013 19:12, Buland Kumar Singh a écrit :
>
>
>
> Hi Patrick,
>
> The following links may also be helpful to understand gdb and
> it's usage for application core analysis.
>
> http://web.eecs.umich.edu/~sugih/pointers/gdb_core.html
> https://sourceware.org/gdb/onlinedocs/gdb/
>
> -- BKS
>
>
> On 17 December 2013 21:36, Patrick Agrain < patrick.agrain(a)alcatel-lucent.com
>> wrote:
>
>
> Hello all,
>
> Now that we have dumped the kernel stack, I'm intesresting in the user
> process from which we came just before the 'panic'.
> Googling around, I found mention of the 'gcore' extension.
>
> I compiled version 1.22 and installed it.
> Using it on crash 6.1.0-1.el6, I get a file core.845.bash on process 'bash'
> (in which I trigger a kernel panic) :
>
>
>
> crash> gcore -v 1 845
> gcore: Opening file core.845.bash ...
> gcore: done.
> gcore: Writing ELF header ...
> gcore: done.
> gcore: Retrieving and writing note information ...
> gcore: done.
> gcore: Writing PT_NOTE program header ...
> gcore: done.
> gcore: Writing PT_LOAD program headers ...
> gcore: done.
> gcore: Writing PT_LOAD segment ...
> gcore: PT_LOAD[0]: 8048000 - 8048000
> gcore: PT_LOAD[1]: 80e2000 - 80e9000
> gcore: PT_LOAD[2]: 80e9000 - 80ed000
> gcore: PT_LOAD[3]: 94a2000 - 94d1000
> gcore: PT_LOAD[4]: b7374000 - b7374000
> gcore: PT_LOAD[5]: b7375000 - b7376000
> gcore: PT_LOAD[6]: b7376000 - b7377000
> gcore: PT_LOAD[7]: b7377000 - b7377000
> gcore: PT_LOAD[8]: b737e000 - b737e000
> gcore: PT_LOAD[9]: b737f000 - b737f000
> gcore: PT_LOAD[10]: b73bb000 - b73bb000
> gcore: PT_LOAD[11]: b75bb000 - b75bb000
> gcore: PT_LOAD[12]: b75c7000 - b75c8000
> gcore: PT_LOAD[13]: b75c8000 - b75c9000
> gcore: PT_LOAD[14]: b75c9000 - b75ca000
> gcore: PT_LOAD[15]: b75ca000 - b75ca000
> gcore: PT_LOAD[16]: b7756000 - b7758000
> gcore: PT_LOAD[17]: b7758000 - b7759000
> gcore: PT_LOAD[18]: b7759000 - b775c000
> gcore: PT_LOAD[19]: b775c000 - b775c000
> gcore: PT_LOAD[20]: b775f000 - b7760000
> gcore: PT_LOAD[21]: b7760000 - b7761000
> gcore: PT_LOAD[22]: b7761000 - b7761000
> gcore: PT_LOAD[23]: b7764000 - b7765000
> gcore: PT_LOAD[24]: b7769000 - b776a000
> gcore: PT_LOAD[25]: b776a000 - b776b000
> gcore: PT_LOAD[26]: b776b000 - b776b000
> gcore: PT_LOAD[27]: b7789000 - b778a000
> gcore: PT_LOAD[28]: b778a000 - b778b000
> gcore: PT_LOAD[29]: bfd07000 - bfd1d000
> gcore: done.
> Saved core.845.bash
> crash>
>
> So far, so good... But
>
> Question: Are there anywhere some hints about how to use this core.<pid> file
> ?
>
> Thanks in advance.
> Regards,
> Patrick Agrain
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
>
>
> --
> BKS
>
>
> --
> Crash-utility mailing list Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
9:32 a.m.
New subject: [SOLVED] Re: About the use of 'gcore'
Hello everybody,
This small mail to close this thread.
The use of gcore in relationship with the 'crash' utility allow us to
debug from kernel space down (up ?) to user space.
It is worth to use it.
Thanks to all for your contribution.
Sincerely,
Patrick Agrain
Le 15/01/2014 08:39, Patrick Agrain a écrit :
Hello everybody,
After complete re-installation of the machine, I'm back on stage.
Dave, to answer to your question, on gdb, the "bt" shows following:
(gdb) bt
#0 0xb776a416 in __kernel_vsyscall ()
#1 0xb7698b43 in __write_nocancel () from /lib/libc.so.6
#2 0xb76343b4 in _IO_new_file_write () from /lib/libc.so.6
#3 0xb7635c90 in _IO_new_do_write () from /lib/libc.so.6
#4 0xb7634e80 in _IO_new_file_overflow () from /lib/libc.so.6
#5 0xb7637d2a in __overflow () from /lib/libc.so.6
#6 0xb76312b5 in putc () from /lib/libc.so.6
#7 0x0809b64b in echo_builtin ()
So your guess was correct, most of the functions are from libc. Also
stated with the 'vm' command.
Performing again the whole process, I think that I now know where I
was wrong.
The error message at the origin of this thread was due to the fact
that I do not use the correct "bash" binary when launching 'gdb'. I
made a mismatch between the "bash" on the crashed system and the
"bash" on the machine where I analyze the crash.
Using the correct "bash" allows to dump the memory and disassemble the
code without problem.
BTW, using the 'vm' and 'vtop' command (Thanks Dave for the tip)
offers interesting information that I will explore later.
For now, I wrote a user-space program that will trigger the page
fault, so that I can more easily make the relationship between
assembly and C language.
To be followed.
Patrick Agrain
Le 06/01/2014 22:08, Dave Anderson a écrit :
as evidenced by the fact that you were able to at least read stack address
bfd1b6d8 from gdb. But it's not clear -- after you walked through the
user stack -- that the dumpfile contains the text page that you're trying
to disassemble. I don't know why it wouldn't be available given that
it's a "hot" code path, and shouldn't be swapped out or otherwise not
mapped.
I'm also presuming that your user-stack walk-through is correct.
When you enter gdb, what does the "bt" command show?
Dave
> ----- Original Message -----
>> Hello all,
>>
>> First of all, I wish a Happy New Year (with less crash, but still enhanced
>> tools...)
>>
>> Thanks for the links, they were very useful.
>> I dig further in the way of analyzing the User Space, but it seems that I'm
>> linked to a dead-end way.
>> Below is a snapshot of kernel / userland stack dump.
>>
>> What I've done :
>> - Crash is triggered by a page fault inside a kernel module (write 0 in
>> 0xFFFFFFFF, classic).
>> - Using gcore to create the 'core.<pid>.bash (which is the user task
running
>> at time of crash).
> I'm curious as to how the bash task was related to the module crash?
> Did the bash task write to a procfs interface that the module created
> to then generate the "write 0 to 0xFFFFFFFF"? Does the crash utility
> indicate that the bash task is the panic task? And if so, what does
> its "bt" show? (i.e., the kernel-mode backtrace)
>
>> - Evaluating an EBP (between { }) chaining value (hypothesis), EIP value
>> (between [ ]) is then just pushed beside
>>
>> The purpose of this study is to find a method to analyze futur crashes from
>> kernel space down to user space applications.
>>
>> Do you have an idea about the cause of this non-dumping of the memory in
>> user-space ?
>> Should I use other extension as 'gcore' ?
>>
>> Thank in advance.
>> Best regards,
>> Patrick Agrain
>>
>>
>> -------
>> ===============================================================================
>> --------------------- Go down into User Space Territory
>> -----------------------
>>
>> Last pt_regs of kernel stack is:
>> | pt_regs
>> 00000001 094a5408 00000003 ..~......TJ..... | bx cx dx
>> c2699fc0: 00000003 094a5408 bfd1b704 00000004 .....TJ......... | si di bp ax
>> c2699fd0: 0000007b ffff007b c07e0000 00000033 {...{.....~.3... | ds es fs gs
>> c2699fe0: 00000004 b776a416 00000073 00000246 ......v.s...F... | orig_eax ip
>> cs flags
>> c2699ff0: bfd1b6d8 0000007b | sp ss
>> v cccccccc cccccccc ....{........... | padding
>> |
>> |----------------------------------------------------------------|
>> |
>> (gdb) x/32xw 0xbfd1b680 |
>> 0xbfd1b680: 0xbfd1b6d0 0x0000000f 0x094b4568 0x080c90b9 |
>> 0xbfd1b690: 0x094b4568 0x080cd160 0x00001936 0x00000001 |
>> 0xbfd1b6a0: 0x094ab9c8 0x00000000 0x094b4b48 0xbfd1b7c8 |
>> 0xbfd1b6b0: 0x080ce9e8 0x094b4b48 0x094b4b48 0xbfd1b728 |
>> 0xbfd1b6c0: 0x094aed28 0x00000020 0x00000000 0x00000070 |
>> 0xbfd1b6d0: 0x094b4588 0x080cc080 |
>> 0xb7698b43 <--|
>> 0xb7757ff4
>> 0xbfd1b6e0: 0xb76343b4 0x00000001 0x094a5408 0x00000003
>> 0xbfd1b6f0: 0xb77584e0 0x080cc080 0xbfd1b728 0xb77584e0
>>
>> |------------------------------------------ Hypothesis : this is an EBP
>> |value...
>> v
>> 0xbfd1b700: 0x00000003 {0xbfd1b72c} [0xb7635c90] 0xb77584e0
>> 0xbfd1b710: 0x094a5408 0x00000003 0x094b4b48 0xbfd1b7c8
>> 0xbfd1b720: 0xb7757ff4 0xb77584e0 0x0000000a {0xbfd1b750}
>> 0xbfd1b730: [0xb7634e80] 0xb77584e0 0x094a5408 0x00000003
>> 0xbfd1b740: 0x0000000a 0xb7757ff4 0xb77584e0 0x0000000a
>> 0xbfd1b750: {0xbfd1b768} [0xb7637d2a] 0xb77584e0 0x0000000a
>> 0xbfd1b760: 0xb7757ff4 0xb77584e0 {0xbfd1b788} [0xb76312b5] >-|
>> 0xbfd1b770: 0xb77584e0 0x0000000a 0xb75c9940 0x094a3e48 |
>> 0xbfd1b780: 0x00000001 0x00000000 0x00000000 0x0809b64b |
>> |
>> Disassemble Try: EIP@0xb76312b5
>> <---------------------------------------------|
>> (gdb) disassemble 0xb7631200, 0xb7631300
>> Dump of assembler code from 0xb7631200 to 0xb7631300:
>> 0xb7631200: Cannot access memory at address 0xb7631200
>> (gdb)
>> ----------
> Anyway, I'm guessing that the 0xb76312b5 IP address is in some
> library, probably libc? If you do a "vm" on the active bash task
> from within the crash utility, you will see where it comes from.
> Try reading the user-space address from the crash utility to see
> if it was available to copy to the core.<pid>.bash file, i.e.,
> try this command:
>
> crash> rd -u 0xb76312b5
>
> The command above presumes that you are in the context of the
> "bash" task while running crash. (i.e., if you enter "set"
alone,
> it shows that particular task)
>
> Dave
>
>
>> Le 17/12/2013 19:12, Buland Kumar Singh a écrit :
>>
>>
>>
>> Hi Patrick,
>>
>> The following links may also be helpful to understand gdb and
>> it's usage for application core analysis.
>>
>> http://web.eecs.umich.edu/~sugih/pointers/gdb_core.html
>> https://sourceware.org/gdb/onlinedocs/gdb/
>>
>> -- BKS
>>
>>
>> On 17 December 2013 21:36, Patrick Agrain <patrick.agrain(a)alcatel-lucent.com
>>> wrote:
>>
>> Hello all,
>>
>> Now that we have dumped the kernel stack, I'm intesresting in the user
>> process from which we came just before the 'panic'.
>> Googling around, I found mention of the 'gcore' extension.
>>
>> I compiled version 1.22 and installed it.
>> Using it on crash 6.1.0-1.el6, I get a file core.845.bash on process
'bash'
>> (in which I trigger a kernel panic) :
>>
>>
>>
>> crash> gcore -v 1 845
>> gcore: Opening file core.845.bash ...
>> gcore: done.
>> gcore: Writing ELF header ...
>> gcore: done.
>> gcore: Retrieving and writing note information ...
>> gcore: done.
>> gcore: Writing PT_NOTE program header ...
>> gcore: done.
>> gcore: Writing PT_LOAD program headers ...
>> gcore: done.
>> gcore: Writing PT_LOAD segment ...
>> gcore: PT_LOAD[0]: 8048000 - 8048000
>> gcore: PT_LOAD[1]: 80e2000 - 80e9000
>> gcore: PT_LOAD[2]: 80e9000 - 80ed000
>> gcore: PT_LOAD[3]: 94a2000 - 94d1000
>> gcore: PT_LOAD[4]: b7374000 - b7374000
>> gcore: PT_LOAD[5]: b7375000 - b7376000
>> gcore: PT_LOAD[6]: b7376000 - b7377000
>> gcore: PT_LOAD[7]: b7377000 - b7377000
>> gcore: PT_LOAD[8]: b737e000 - b737e000
>> gcore: PT_LOAD[9]: b737f000 - b737f000
>> gcore: PT_LOAD[10]: b73bb000 - b73bb000
>> gcore: PT_LOAD[11]: b75bb000 - b75bb000
>> gcore: PT_LOAD[12]: b75c7000 - b75c8000
>> gcore: PT_LOAD[13]: b75c8000 - b75c9000
>> gcore: PT_LOAD[14]: b75c9000 - b75ca000
>> gcore: PT_LOAD[15]: b75ca000 - b75ca000
>> gcore: PT_LOAD[16]: b7756000 - b7758000
>> gcore: PT_LOAD[17]: b7758000 - b7759000
>> gcore: PT_LOAD[18]: b7759000 - b775c000
>> gcore: PT_LOAD[19]: b775c000 - b775c000
>> gcore: PT_LOAD[20]: b775f000 - b7760000
>> gcore: PT_LOAD[21]: b7760000 - b7761000
>> gcore: PT_LOAD[22]: b7761000 - b7761000
>> gcore: PT_LOAD[23]: b7764000 - b7765000
>> gcore: PT_LOAD[24]: b7769000 - b776a000
>> gcore: PT_LOAD[25]: b776a000 - b776b000
>> gcore: PT_LOAD[26]: b776b000 - b776b000
>> gcore: PT_LOAD[27]: b7789000 - b778a000
>> gcore: PT_LOAD[28]: b778a000 - b778b000
>> gcore: PT_LOAD[29]: bfd07000 - bfd1d000
>> gcore: done.
>> Saved core.845.bash
>> crash>
>>
>> So far, so good... But
>>
>> Question: Are there anywhere some hints about how to use this core.<pid>
file
>> ?
>>
>> Thanks in advance.
>> Regards,
>> Patrick Agrain
>>
>> --
>> Crash-utility mailing list
>> Crash-utility(a)redhat.com
>> https://www.redhat.com/mailman/listinfo/crash-utility
>>
>>
>>
>> --
>> BKS
>>
>>
>> --
>> Crash-utility mailing listCrash-utility(a)redhat.com
>> https://www.redhat.com/mailman/listinfo/crash-utility
>>
>>
>> --
>> Crash-utility mailing list
>> Crash-utility(a)redhat.com
>> https://www.redhat.com/mailman/listinfo/crash-utility
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
3812
days inactive
3858
days old
devel@lists.crash-utility.osci.io
16 comments
4 participants
participants (4)
-
Buland Kumar Singh -
Dave Anderson -
Louis Bouchard -
Patrick Agrain