[RFC v2] Use register value in elf note NT_PRSTATUS to do backtrace
by Wang Chao
Hi Dave and list,
I guess you still remember that a few weeks ago Wen has sent a patch which intends
to use register values in NT_PRSTATUS notes for backtracing if no panic task was
found. Thanks for your review and the suggestions are very useful.
However, Wen was busy with other work for these days, so I'll continue with this
work and now the 2rd version patch is attached.
v2: Address review feedbacks.
1) Set up a flag in pc->flags2 and it's determined during the diskdump file init
procedure.
2) Seperate code according to the that flag.
3) Also, we've done some tests on dumpfile of xen kernel and the trouble described
in the previous mail was gone.
So we're looking forward to your feedback and if you still have any problems with
it, please tell us.
Thanks,
Wang Chao
13 years, 6 months
[RESEND] Re: Check _PAGE_PROTNONE bit in uvtop()
by Dave Anderson
I'm resending this response to Daisuke's patch because the
mailing list seems to have been confused by the combination
of his original patch and my response to it. The mailing
list archive has "lost" my response, and shows this instead:
https://www.redhat.com/archives/crash-utility/2011-April/msg00032.html
which is actually the end of Daisuke's patch posting. Weird...
Anyway, here's a cut-and-paste of my actual response, which
was sent out individually OK, but I'm guessing that the "digest"
version will show the bogus response above.
----- Original Message -----
> Currently, some of uvtop() functions on x86, x86_64 and ia64 lacks
> checking whether in pte _PAGE_PROTNONE bit is set or not. The flag is
> set to pte when specifying to the corresponding memory space PROT_NONE
> using mprotect(), and then _PAGE_PRESENT bit in the pte is unset in
> order to make hardware cause segmentation fault
> intensionally. Important is the fact that besides _PAGE_PRESENT set,
> _PAGE_PROTNONE indicates the page is present (not swapped out).
>
> The architectures fixed are x86, x86_64 and ia64 except for Xen.
>
> The patchset based on crash 5.1.4 consists of 5 patches, which are
> divided into the ones extending machine dependent data on x86 and
> x86_64, and the ones actually adding _PAGE_PROTNONE checking into each
> uvtop().
>
> defs.h | 6 ++++--
> ia64.c | 6 +++---
> x86.c | 10 ++++++++--
> x86_64.c | 11 +++++++++--
> 4 files changed, 24 insertions(+), 9 deletions(-)
Daisuke,
I have seen this "unknown swap location" on rare occasions when running
"vm -p" on the "ntpd" daemon. I had determined that the task had done an
mprotect(PROT_NONE), but never got around to fixing it. Thanks for
addressing this.
Also, I will add a display of the new page_protnone field in both the
x86_dump_machdep_table() and x86_64_dump_machdep_table() functions.
Queued for the next release.
Thanks,
Dave
13 years, 7 months
simple question
by Koornstra, Reinoud
Hi Everyone,
How're you?
Loading kernel core dumps in gdb requires to have a compatible gdb for version of gcc which you compiled the source with.
Otherwise you see "is not a core dump: File format not recognized".
Crash however, has never trouble and loads the core dumps always in without trouble.
Forgive my ignorance, but how did you make it work where gdb cannot load it?
Btw, crash is a life saver when it comes to debugging, at least for me. :)
Thanks,
Reinoud.
13 years, 7 months
Re: [Crash-utility] x86_64 limit of 454 cpu's?
by tachibana＠mxm.nes.nec.co.jp
Hi,
On Fri, 15 Apr 2011 12:13:51 -0400 (EDT)
Dave Anderson <anderson(a)redhat.com> wrote:
>
>
> ----- Original Message -----
> > (4/15/2011 09:04), Dave Anderson wrote:
> > >
> > >
> > > ----- Original Message -----
> > >> Hi Dave, and company,
> > >>
> > >> I get this error trying to open a dump of a large system:
> > >>
> > >> crash: compressed kdump: invalid nr_cpus value: 640
> > >> crash: vmcore: not a supported file format
> > >>
> > >> The message is from diskdump.c:
> > >> if (sizeof(*header) + sizeof(void *) * header->nr_cpus> block_size
> > >> ||
> > >> header->nr_cpus<= 0) {
> > >> error(INFO, "%s: invalid nr_cpus value: %d\n",
> > >>
> > >> block_size is the page size of 4096
> > >> struct disk_dump_header looks like 464 bytes
> > >> void * is 8
> > >> So it looks like 454 is the maximum number of cpus.
> > >> 464 + 454*8 -> 4096
> > >>
> > >> Is this intentional?
> > >> It looks like a restriction that no one ever complained about. But
> > >> there
> > >> are systems (Altix UV) with 2048 cpu's.
> > >>
> > >> Is there an easy fix?
> > >>
> > >> -Cliff
> > >
> > > To be honest, I don't know, I didn't design or write that code.
> >
> > Yes, this is intentional for RHEL4/diskdump. In the RHEL4 kernel,
> > disk_dump_header is defined as follows.
> >
> > struct disk_dump_header {
> > char signature[8]; /* = "DISKDUMP" */
> > (snip)
> > int nr_cpus; /* Number of CPUs */
> > struct task_struct *tasks[NR_CPUS];
> > };
> >
> > And maximum logical CPUs of RHEL4 are 32(x86) or 64(x86_64) so
> > this does not cause any problem.
> >
> > On the other hands, as you and Dave said, this causes limitation
> > problem
> > in RHEL5/RHEL6 kernel. But as far as I know, makedumpfile does not use
> > this "tasks" member, so we can skip here.
> >
> > if (is_diskdump?)
> > if (sizeof(*header) + sizeof(void *) * header->nr_cpus> block_size ||
> > header->nr_cpus<= 0) {
> > error(INFO, "%s: invalid nr_cpus value: %d\n",
> > goto err;
> > }
> >
> > Something like this. Dave, Ohmichi-san, what do you think?
> >
> > Thanks,
> > Takao Indoh
>
> Looking at a couple sample compressed kdumps, it does appear that
> they do not set up the tasks[] array, and that the sub_header still
> starts at the "header->block_size". That being the case, your
> proposal looks like a nice, simple fix!
>
> Cliff, can you enclose that piece of code with something like:
>
> if (DISKDUMP_VALID()) {
> if (sizeof(*header) + sizeof(void *) * header->nr_cpus > block_size ||
> header->nr_cpus <= 0) {
> error(INFO, "%s: invalid nr_cpus value: %d\n",
> DISKDUMP_VALID() ? "diskdump" : "compressed kdump",
> header->nr_cpus);
> goto err;
> }
> }
>
> I suppose it should still check for (header->nr_cpus <= 0), but I'll
> let Ken'ichi confirm that.
I think that it is good to check the format of the header though
makedumpfile sets header->nr_cpus of course.
So I think that the following code is better.
if ((DISDUMP_VALID() &&
sizeof(*header) + sizeof(void *) * header->nr_cpus > block_size) ||
header->nr_cpus <= 0) {
error(INFO, "%s: invalid nr_cpus value: %d\n",
DISKDUMP_VALID() ? "diskdump" : "compressed kdump",
header->nr_cpus);
goto err;
}
And,
- makedumpfile doesn't use tasks array as Indoh-san said.
- I don't think that makedumpfile malfunctions for CPUs more than 454.
Thanks,
tachibana
>
> Thanks Takao,
> Dave
>
> >
> > >
> > > And you're right, although dumpfiles with that many cpus are highly
> > > unusual, but looking at the code, it certainly does appear that the
> > > disk_dump_header plus the task pointers for each cpu must fit in a
> > > "block_size", or page size, and that the sub_header is the first
> > > item
> > > following the contents of the first page:
> > >
> > > ---
> > > ^ disk_dump_header
> > > | task on cpu 0
> > > page ...
> > > | task on cpu x-1
> > > V task on cpu x
> > > ---
> > > sub_header
> > > bitmap
> > > remaining stuff...
> > >
> > > Since your dump is presumably a compressed kdump, I'm wondering
> > > what the makedumpfile code did in your case? Did it round up the
> > > location of the sub_header to a page boundary?
> > >
> > > I've cc'd this to Ken'ichi Ohmichi (makedumpfile), and to Takao
> > > Indoh
> > > (original compressed diskdump) for their input.
> > >
> > > Thanks,
> > > Dave
> > >
> > >
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
13 years, 7 months
x86_64 limit of 454 cpu's?
by Cliff Wickman
Hi Dave, and company,
I get this error trying to open a dump of a large system:
crash: compressed kdump: invalid nr_cpus value: 640
crash: vmcore: not a supported file format
The message is from diskdump.c:
if (sizeof(*header) + sizeof(void *) * header->nr_cpus > block_size ||
header->nr_cpus <= 0) {
error(INFO, "%s: invalid nr_cpus value: %d\n",
block_size is the page size of 4096
struct disk_dump_header looks like 464 bytes
void * is 8
So it looks like 454 is the maximum number of cpus.
464 + 454*8 -> 4096
Is this intentional?
It looks like a restriction that no one ever complained about. But there
are systems (Altix UV) with 2048 cpu's.
Is there an easy fix?
-Cliff
13 years, 7 months
Re: [Crash-utility] DD image
by Dave Anderson
----- Original Message -----
> Hi Dave,
>
> Thanks for your feedback , actually what I meant by minimizing the
> footprint that to not preform any action which requires instillation
> for any tool in order to image the memory. that to preserve the
> volatile data and not to overwrite information in memory such as
> terminated and cached processes. snap.so is good solution for live
> system but it first requires the instillation of crash in the
> investigated system which is not forensic sound. dd is small tool that
> comes with the most modern Linux destructions and forensic toolkits as
> it does not require instillation. if crash supports dd image then
> crach utility will be not only debugger but also forensic tool.
>
> thanks,
> Amer
OK, so you *are* suggesting that the dumpfile would be created
something like this:
$ dd if=/dev/mem of=memory-image bs=<page-size>
What architecture is this for?
And can you confirm that the /dev/mem read_mem() function will
proceed without any of its error scenarios occurring before the
last page of physical memory is read?
There are these possible failures:
if (!valid_phys_addr_range(p, count))
return -EFAULT;
if (!range_is_allowed(p >> PAGE_SHIFT, count))
return -EPERM;
ptr = xlate_dev_mem_ptr(p);
if (!ptr)
return -EFAULT;
remaining = copy_to_user(buf, ptr, sz);
unxlate_dev_mem_ptr(p, ptr);
if (remaining)
return -EFAULT;
First, the valid_phys_addr_range() restricts /dev/mem to
high_memory, or 896MB maximum on 32-bit architectures.
The crash utility will still initialize, but several
commands will fail if memory over that 896MB threshold
are required.
But more importantly, the xlate_dev_mem_ptr() call is the one
that could possibly trip you up if the architecture is x86 or x86_64.
I don't have a machine on-hand to test that because RHEL/Fedora
kernels apply CONFIG_STRICT_DEVMEM, so /dev/mem is useless for
that purpose.
In any case, if the output file is page-for-page copy
of physical memory, then creating support for it would
be fairly easy. But I really don't want to expend any effort
creating support for such a file format given the potential
problems with the use of /dev/mem.
On the other hand, it would also be fairly easy to create
a small utility function that simply pre-pends an ELF header
to the dumpfile -- one which has a single PT_LOAD section
that describes the physical memory as one large chunk.
For this simple format, you could take the snap.c extension
module's generate_elf_header() function, have it create a
an ELF header with just one PT_LOAD segment, and fold it
into a standalone program. It has support for x86, x86_64,
ppc64 and ia64.
Dave
13 years, 7 months
Re: [Crash-utility] DD image
by takuo.koguchi.sw＠hitachi.com
Hi Dave,
Crash utility support for such a raw dumpfile would be really useful for some embedded devices.
Such device typically have no storage resource to write the dumpfile in the supported format, but another CPU on the system can take out the physical memory contents to a connected debugger PC. In this case, only raw dumpfile is available since the latter CPU do not have the knowledge of the crashed kernel.
Writing a small utility which converts a raw dump to one of the supported format might be an idea. But it probably requires the information from vmlinux.
So it seems natural to me that crash utility should support raw dumpfile by itself.
Best Regard,
Takuo Koguchi
>
>----- Original Message -----
>> Hi ,
>>
>>
>> recently, some forensic research suggested that utilizing Crash
>> utility as independent solution to parse Linux memory dump in order to
>> extract forensic artifacts. but in real forensic cases where there is
>> need for minimizing the footprint on the comprised system, the
>> forensic analyst would perform only one action, which is physical
>> memory capture to minimize the footprint with dd. I just wonder if
>> there any chance that Crach utility would support dd image.
>>
>> Thanks,
>> Amer
>
>Certainly there is no support for such a raw dumpfile format.
>
>But I don't really understand what you mean by saying that the
>use of dd "would minimize the footprint"? I presume that you
>are asking whether you could do something like this on a live
>system?:
>
> $ dd if=/dev/mem of=memory-image
> $ crash vmlinux memory-image
>
>Theoretically it could be done, presuming that the read_mem()
>function in the /dev/mem driver would never fail until it reached
>the end of physical memory, i.e., would create an exact page-by-page
>copy of all physical pages from 0 to the end of physical memory.
>
>But if that's the case, and you can run crash on the system that
>you want to dump, try the "snap.so" extension module that comes
>with the crash utility source package. It creates a dumpfile
>while running on a live system, in an ELF format that crash
>understands.
>
>Dave
>
>--
>Crash-utility mailing list
>Crash-utility(a)redhat.com
>https://www.redhat.com/mailman/listinfo/crash-utility
>
13 years, 7 months
missing FP/SP in a crash dump
by Alok Aggarwal
Greetings,
I'm trying to debug a crash dump with the 'crash'
utility but I'm running into the stack pointer and
frame pointer being absent from the stack backtrace.
A snippet of the dump is:
..
#4 [ffff88084c969ef0] vfs_write at ffffffff8116d098
ffff88084c969ef8: ffff88084c969f78 ffffffff810d42b2
ffff88084c969f08: ffff88085b280660 ffff880858d5f240
ffff88084c969f18: 00007f43f3d9d000 0000000000000002
ffff88084c969f28: ffff88084c969f78 ffffffff8116dad1
#5 [ffff88084c969f30] sys_write at ffffffff8116dad1
ffff88084c969f38: ffff88084c969f78 00007f43f3d9d000
ffff88084c969f48: 0000000000000000 0000000000000002
ffff88084c969f58: 000000324b779780 0000000000000002
ffff88084c969f68: 000000324b779780 0000000000000002
ffff88084c969f78: 00007f43f3d9d000 ffffffff81013172
..
I'm doing this on RHEL6, specifically -
Linux cache7 2.6.32-71.14.1.el6.x86_64 #1 SMP Wed Jan 5 17:01:01 EST 2011 x86_64
x86_64 x86_64 GNU/Linux
Any ideas why my dump might be missing the info about
SP/FP?
Thanks,
Alok
13 years, 7 months
DD image
by Amer Aljaedi
Hi ,
recently, some forensic research suggested that utilizing Crash utility as
independent solution to parse Linux memory dump in order to extract
forensic artifacts. but in real forensic cases where there is need for
minimizing the footprint on the comprised system, the forensic analyst
would perform only one action, which is physical memory capture to minimize
the footprint with dd. I just wonder if there any chance that Crach utility
would support dd image.
Thanks,
Amer
13 years, 7 months
