6:02 p.m.
Hello,
For some weeks I've developed gcore subcommand for crash utility which
provides process coredump feature for crash kernel dump, strongly
demanded by users who want to investigate user-space applications
contained in kernel crash dump.
I've now finished making a prototype version of gcore and found out
what are the issues to be addressed intensely. Could you give me any
comments and suggestions on this work?
Hello Daisuke,
As I mentioned in my previous email re: cpu numbering, I am currently
on vacation, and cannot spend much time looking at this issue until
I get back on August 9th.
However, I think that this could be a useful feature, and I did
take a quick look at how it could be done several months ago when
it was brought up on this mailing list. However, as you discovered,
I also noted that the user-space core dump code in the kernel has
undergone significant changes over time, and so the implemetation
by the crash utility would have to adapt to the kernel data structures
used by the various kernel versions. And because of that, I don't
want to put it into the base crash binary, but rather it should be
maintained as one or more extension modules, which can be located
in the "extensions" subdirectory in the crash source package, as well
as stored in the "extensions" web page link from the crash "people"
web site.
It is quite simple to re-adapt your patch as an extension module.
Check the "snap.c" and "snap.mk" files in the extensions subdirectory
as templates for your "gcore" command.
As to the other questions below, I will get back to you after
August 9th.
Thanks,
Dave
> Motivation
> ==========
>
> It's a relatively familiar technique that in a cluster system a
> currently running node triggers crash kernel dump mechanism when
> detecting a kind of a critical error in order for the running, error
> detecting server to cease as soon as possible. Concequently, the
> residual crash kernel dump contains a process image for the erroneous
> user application. At the case, developpers are interested in user
> space, rather than kernel space.
>
> There's also a merit of gcore that it allows us to use several
> userland debugging tools, such as GDB and binutils, in order to
> analyze user space memory.
>
>
> Current Status
> ==============
>
> I confirm the prototype version runs on the following configuration:
>
> Linux Kernel Version: 2.6.34
> Supporting Architecture: x86_64
> Crash Version: 5.0.5
> Dump Format: ELF
>
> I'm planning to widen a range of support as follows:
>
> Linux Kernel Version: Any
> Supporting Architecture: i386, x86_64 and IA64
> Dump Format: Any
>
>
> Issues
> ======
>
> Currently, I have issues below.
>
> 1) Retrieval of appropriate register values
>
> The prototype version retrieves register values from a _wrong_
> location: a top of the kernel stack, into which register values are
> saved at any preemption context switch. On the other hand, the
> register values that should be included here are the ones saved at
> user-to-kernel context switch on any interrupt event.
>
> I've yet to implement this. Specifically, I need to do the following
> task from now.
>
> (1) list all entries from user-space to kernel-space execution path.
>
> (2) divide the entries according to where and how the register
> values from user-space context are saved.
>
> (3) compose a program that retrieves the saved register values from
> appropriate locations that is traced by means of (1) and (2).
>
> Ideally, I think it's best if crash library provides any means of
> retrieving this kind of register values, that is, ones saved on
> various stack frames. Is there such a plan to do?
>
>
> 2) Getting a signal number for a task which was during core dump
> process at kernel crash
>
> If a target task is halfway of core dump process, it's better to know
> a signal number in order to know why the task was about to be core
> dumped.
>
> Unfortunately, I have no choice but backtrace the kernel stack to
> retrieve a signal number saved there as an argument of, for example,
> do_coredump().
>
>
> 3) Kernel version compatibility
>
> crash's policy is to support all kernel versions by the latest crash
> package. On the other hand, the prototype is based on kernel 2.6.34.
> This means more kernel versions need to be supported.
>
> Well, the question is: to what versions do I need to really test in
> addition to the latest upstream kernel? I think it's practically
> enough to support RHEL4, RHEL5 and RHEL6.
>
>
> Build Instruction
> =================
>
> $ tar xf crash-5.0.5.tar.gz
> $ cd crash-5.0.5/
> $ patch -p 1 < gcore.patch
> $ make
>
>
> Usage
> =====
>
> Use help subcommand of crash utility as ``help gcore''.
>
>
> Attached File
> =============
>
> * gcore.patch
>
> A patch implementing gcore subcommand for crash-5.0.5.
>
> The diffstat output is as follows.
>
> $ diffstat gcore.patch
> Makefile | 10 +-
> defs.h | 15 +
> gcore.c | 1858
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> gcore.h | 639 ++++++++++++++++++++
> global_data.c | 3 +
> help.c | 28 +
> netdump.c | 27 +
> tools.c | 37 ++
> 8 files changed, 2615 insertions(+), 2 deletions(-)
>
> --
> HATAYAMA Daisuke
> d.hatayama(a)jp.fujitsu.com
>
8:14 a.m.
New subject: [RFC] gcore subcommand: a process coredump feature
Hello Dave,
Thank you for your comment.
From: anderson(a)prospeed.net
Subject: Re: [Crash-utility] [RFC] gcore subcommand: a process coredump feature
Date: Mon, 2 Aug 2010 19:02:23 -0400 (EDT)
>
> Hello,
>
> For some weeks I've developed gcore subcommand for crash utility which
> provides process coredump feature for crash kernel dump, strongly
> demanded by users who want to investigate user-space applications
> contained in kernel crash dump.
>
> I've now finished making a prototype version of gcore and found out
> what are the issues to be addressed intensely. Could you give me any
> comments and suggestions on this work?
Hello Daisuke,
As I mentioned in my previous email re: cpu numbering, I am currently
on vacation, and cannot spend much time looking at this issue until
I get back on August 9th.
However, I think that this could be a useful feature, and I did
take a quick look at how it could be done several months ago when
it was brought up on this mailing list. However, as you discovered,
I hear for the first time that the same kind of proposal was already
proposed previously on this mailing list. I try to find it to compare
with mine.
I also noted that the user-space core dump code in the kernel has
undergone significant changes over time, and so the implemetation
by the crash utility would have to adapt to the kernel data structures
used by the various kernel versions. And because of that, I don't
want to put it into the base crash binary, but rather it should be
maintained as one or more extension modules, which can be located
in the "extensions" subdirectory in the crash source package, as well
as stored in the "extensions" web page link from the crash "people"
web site.
I agree basically, but I think a main stream of gcore can steadily be
shared among the ones for different kernel versions, since dependent
kernel data structures around there are mm and mmap members of
task_struct and vm_* members of mm_struct only. So, I think it
possible to keep the main stream in binary and make only
kernel-version specific sub-programs to gather kinds of note
information be distributed as shared libraries.
It is quite simple to re-adapt your patch as an extension module.
Check the "snap.c" and "snap.mk" files in the extensions
subdirectory
as templates for your "gcore" command.
As to the other questions below, I will get back to you after
August 9th.
Thanks, I'm waiting for your further comments.
>
> Thanks,
> Dave
>
>
>> Motivation
>> ==========
>>
>> It's a relatively familiar technique that in a cluster system a
>> currently running node triggers crash kernel dump mechanism when
>> detecting a kind of a critical error in order for the running, error
>> detecting server to cease as soon as possible. Concequently, the
>> residual crash kernel dump contains a process image for the erroneous
>> user application. At the case, developpers are interested in user
>> space, rather than kernel space.
>>
>> There's also a merit of gcore that it allows us to use several
>> userland debugging tools, such as GDB and binutils, in order to
>> analyze user space memory.
>>
>>
>> Current Status
>> ==============
>>
>> I confirm the prototype version runs on the following configuration:
>>
>> Linux Kernel Version: 2.6.34
>> Supporting Architecture: x86_64
>> Crash Version: 5.0.5
>> Dump Format: ELF
>>
>> I'm planning to widen a range of support as follows:
>>
>> Linux Kernel Version: Any
>> Supporting Architecture: i386, x86_64 and IA64
>> Dump Format: Any
>>
>>
>> Issues
>> ======
>>
>> Currently, I have issues below.
>>
>> 1) Retrieval of appropriate register values
>>
>> The prototype version retrieves register values from a _wrong_
>> location: a top of the kernel stack, into which register values are
>> saved at any preemption context switch. On the other hand, the
>> register values that should be included here are the ones saved at
>> user-to-kernel context switch on any interrupt event.
>>
>> I've yet to implement this. Specifically, I need to do the following
>> task from now.
>>
>> (1) list all entries from user-space to kernel-space execution path.
>>
>> (2) divide the entries according to where and how the register
>> values from user-space context are saved.
>>
>> (3) compose a program that retrieves the saved register values from
>> appropriate locations that is traced by means of (1) and (2).
>>
>> Ideally, I think it's best if crash library provides any means of
>> retrieving this kind of register values, that is, ones saved on
>> various stack frames. Is there such a plan to do?
>>
>>
>> 2) Getting a signal number for a task which was during core dump
>> process at kernel crash
>>
>> If a target task is halfway of core dump process, it's better to know
>> a signal number in order to know why the task was about to be core
>> dumped.
>>
>> Unfortunately, I have no choice but backtrace the kernel stack to
>> retrieve a signal number saved there as an argument of, for example,
>> do_coredump().
>>
>>
>> 3) Kernel version compatibility
>>
>> crash's policy is to support all kernel versions by the latest crash
>> package. On the other hand, the prototype is based on kernel 2.6.34.
>> This means more kernel versions need to be supported.
>>
>> Well, the question is: to what versions do I need to really test in
>> addition to the latest upstream kernel? I think it's practically
>> enough to support RHEL4, RHEL5 and RHEL6.
>>
>>
>> Build Instruction
>> =================
>>
>> $ tar xf crash-5.0.5.tar.gz
>> $ cd crash-5.0.5/
>> $ patch -p 1 < gcore.patch
>> $ make
>>
>>
>> Usage
>> =====
>>
>> Use help subcommand of crash utility as ``help gcore''.
>>
>>
>> Attached File
>> =============
>>
>> * gcore.patch
>>
>> A patch implementing gcore subcommand for crash-5.0.5.
>>
>> The diffstat output is as follows.
>>
>> $ diffstat gcore.patch
>> Makefile | 10 +-
>> defs.h | 15 +
>> gcore.c | 1858
>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> gcore.h | 639 ++++++++++++++++++++
>> global_data.c | 3 +
>> help.c | 28 +
>> netdump.c | 27 +
>> tools.c | 37 ++
>> 8 files changed, 2615 insertions(+), 2 deletions(-)
>>
>> --
>> HATAYAMA Daisuke
>> d.hatayama(a)jp.fujitsu.com
>>
3 a.m.
New subject: [RFC] gcore subcommand: a process coredump feature
HATAYAMA Daisuke wrote:
> However, I think that this could be a useful feature, and I did
> take a quick look at how it could be done several months ago when
> it was brought up on this mailing list. However, as you discovered,
I hear for the first time that the same kind of proposal was already
proposed previously on this mailing list. I try to find it to compare
with mine.
AFAIK many people are interested in this feature and it has been discussed
before but nobody actually offered a patch until now. Last thread I am aware of
is this one:
https://www.redhat.com/archives/crash-utility/2010-February/msg00055.html
Admittedly, that thread is just about extracting an userland backtrace. Maybe
your patch can be adapted to add a "btu" command (or maybe "bt -u") to
extract
an userland backtrace directly?
Thank you very much anyway :)
8:14 a.m.
New subject: [RFC] gcore subcommand: a process coredump feature
From: Adrien Kunysz <adk(a)redhat.com>
Subject: Re: [Crash-utility] [RFC] gcore subcommand: a process coredump feature
Date: Tue, 03 Aug 2010 09:00:56 +0100
HATAYAMA Daisuke wrote:
>> However, I think that this could be a useful feature, and I did
>> take a quick look at how it could be done several months ago when
>> it was brought up on this mailing list. However, as you discovered,
> I hear for the first time that the same kind of proposal was already
> proposed previously on this mailing list. I try to find it to compare
> with mine.
AFAIK many people are interested in this feature and it has been
discussed before but nobody actually offered a patch until now. Last
thread I am aware of is this one:
https://www.redhat.com/archives/crash-utility/2010-February/msg00055.html
Admittedly, that thread is just about extracting an userland
backtrace. Maybe your patch can be adapted to add a "btu" command (or
maybe "bt -u") to extract an userland backtrace directly?
Oh, thanks. It helps a lot.
If I've understood correctly, crash doesn't prepare any means of
handling symbolic information for user-space processes. In this sense,
I think the ``direct backtrace'' impossible.
On the other hand, if extracting user-space process image, one can use
GDB to see the process's backtrace using GDB's bt sub-command.
But for this, it's important to gather correct register values, in
particular, stack pointer register. But unfortunately, the prototype
I've demonstrated is not complete as I've described in the first
entry.
Thanks.
--
HATAYAMA Daisuke
Thank you very much anyway :)
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
10:06 a.m.
New subject: [RFC] gcore subcommand: a process coredump feature
----- "HATAYAMA Daisuke" <d.hatayama(a)jp.fujitsu.com> wrote:
From: Adrien Kunysz <adk(a)redhat.com>
> Admittedly, that thread is just about extracting an userland
> backtrace. Maybe your patch can be adapted to add a "btu" command (or
> maybe "bt -u") to extract an userland backtrace directly?
Oh, thanks. It helps a lot.
If I've understood correctly, crash doesn't prepare any means of
handling symbolic information for user-space processes. In this sense,
I think the ``direct backtrace'' impossible.
On the other hand, if extracting user-space process image, one can use
GDB to see the process's backtrace using GDB's bt sub-command.
Right -- I agree with Daisuke that the whole purpose of this gcore
command is to extract a core file for subsequent use with gdb.
But FWIW, you can load the debuginfo data of a user-space program into a
crash session with the "add-symbol-file" gdb command. The user-space
symbol values are then maintained only in the embedded gdb module, i.e.,
they are not available with the crash "sym" command, for example. But
they can be accessed with commands that in turn use embedded gdb commands.
For example, I've done it in the past for debugging a live crash session
from itself. For example:
# ./crash
crash 5.0.6
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
KERNEL: /usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux
DUMPFILE: /dev/crash
CPUS: 8
DATE: Wed Aug 11 10:43:03 2010
UPTIME: 2 days, 02:20:05
LOAD AVERAGE: 0.13, 0.08, 0.06
TASKS: 252
NODENAME: crash.usersys.redhat.com
RELEASE: 2.6.18-128.el5
VERSION: #1 SMP Wed Dec 17 11:41:38 EST 2008
MACHINE: x86_64 (1995 Mhz)
MEMORY: 1 GB
PID: 24111
COMMAND: "crash"
TASK: ffff81001d88e7e0 [THREAD_INFO: ffff81002912e000]
CPU: 5
STATE: TASK_RUNNING (ACTIVE)
crash>
You have to set/leave the crash context to that of the program you're
interested in, because when accessing user virtual addresses, the current
context's memory space is read.
First, get the starting text address:
crash> vm
PID: 24111 TASK: ffff81001d88e7e0 CPU: 5 COMMAND: "crash"
MM PGD RSS TOTAL_VM
ffff81003e075480 ffff8100032cd000 112640k 191264k
VMA START END FLAGS FILE
ffff810021a99768 400000 95d000 1875 /var/CVS/crash/crash
ffff81001e06c298 b5d000 b7d000 101873 /var/CVS/crash/crash
ffff81001e06c558 b7d000 ce5000 100073
ffff81001e06ce48 15976000 19062000 100073
ffff81001e06c768 3fad600000 3fad61a000 875 /lib64/ld-2.5.so
ffff81001e06c348 3fad81a000 3fad81b000 100871 /lib64/ld-2.5.so
...
And then load the debuginfo data into the crash session at that
user text address:
crash> add-symbol-file ./crash 0x400000
add symbol table from file "./crash" at
.text_addr = 0x400000
Reading symbols from /var/CVS/crash/crash...done.
crash>
And then, for example, the embedded gdb knows about the global
"pc" pointer used by the crash utility to store general context
data:
crash> whatis pc
struct program_context *
crash>
And its pointer value can be determined from "p -u":
crash> p -u pc
$2 = (struct program_context *) 0xb7d420
crash>
And then either displayed with "struct -u":
crash> struct -u program_context 0xb7d420
struct program_context {
program_name = 0x7fffcd91dc24 "crash",
program_path = 0x7fffcd91dc22 "./crash",
program_version = 0x7bc720 "5.0.6",
gdb_version = 0x828e00 "7.0",
prompt = 0x15989a20 "crash> ",
flags = 879609304517639,
namelist = 0x1597a500 "/usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux",
dumpfile = 0x0,
live_memsrc = 0x76f467 "/dev/crash",
system_map = 0x0,
namelist_debug = 0x0,
debuginfo_file = 0x0,
memory_module = 0x77c0f7 "crash",
memory_device = 0x76f467 "/dev/crash",
machine_type = 0x76f460 "X86_64",
editing_mode = 0x76ee85 "vi",
server = 0x0,
server_memsrc = 0x0,
server_namelist = 0x0,
nfd = -1,
mfd = 4,
kfd = -1,
dfd = -1,
confd = -2,
sockfd = 0,
port = 0,
rmfd = 0,
rkfd = 0,
program_pid = 24111,
...
Or the same thing could be done like so:
crash> p -u *pc
$3 = {
program_name = 0x7fffcd91dc24 "crash",
program_path = 0x7fffcd91dc22 "./crash",
program_version = 0x7bc720 "5.0.6",
...
It's an ugly process, but it can be accomplished if necessary.
Dave
1:04 p.m.
New subject: crash: invalid structure member offset
Hi Everyone,
I am trying to read a core file into crash, but I've got bad luck as you can see
below.
Is core file corrupt? It is a vmcore file from a 32 bits kernel that was compiled with
PAE, could that have corrupted things?
Any hints here?
Thanks,
Reinoud.
$ crash System.map-2.6.27 ./vmlinux-2.6.27 ./vmcore
crash 4.0-3.7
Copyright 2002, 2003, 2004, 2005, 2006 Red Hat, Inc.
Copyright 2004, 2005, 2006 IBM Corporation
Copyright 1999-2006 Hewlett-Packard Co
Copyright 2005 Fujitsu Limited
Copyright 2005 NEC Corporation
Copyright 1999, 2002 Silicon Graphics, Inc.
Copyright 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
please wait... (gathering kmem slab cache data)
crash: invalid structure member offset: kmem_cache_s_c_num
FILE: memory.c LINE: 6891 FUNCTION: kmem_cache_init()
[/usr/bin/crash] error trace: 80827a9 => 8095398 => 80aa7ef => 8131e88
/usr/bin/nm: /usr/bin/crash: no symbols
/usr/bin/nm: /usr/bin/crash: no symbols
/usr/bin/nm: /usr/bin/crash: no symbols
/usr/bin/nm: /usr/bin/crash: no symbols
WARNING: Because this kernel was compiled with gcc version 4.1.2, certain
commands or command options may fail unless crash is invoked with
the "--readnow" command line option.
5073
days inactive
5082
days old
devel@lists.crash-utility.osci.io
5 comments
5 participants
participants (5)
-
Adrien Kunysz -
anderson＠prospeed.net -
Dave Anderson -
HATAYAMA Daisuke -
Koornstra, Reinoud