February 2012 - Crash-utility - Crash Utility List Archives

Re: [Crash-utility] question about phys_base

by Dave Anderson

----- Original Message ----- > > > > OK, so then I don't understand what you mean by "may be the same"? > > > > You didn't answer my original question, but if I understand you correctly, > > it would be impossible for the qemu host to create a PT_LOAD segment that > > describes an x86_64 guest's __START_KERNEL_map region, because the host > > doesn't know that what kind of kernel the guest is running. > > Yes. Even if the guest is linux, it is still impossible to do it. Because > the guest maybe in the second kernel. > > qemu-dump walks all guest's page table and collect virtual address and > physical address mapping. If the page is not used by guest, the virtual is set > to 0. I create PT_LOAD according to such mapping. So if the guest is linux, > there may be a PT_LOAD segment that describes __START_KERNEL_map region. > But the information stored in PT_LOAD maybe for the second kernel. If crash > uses it, crash will see the second kernel, not the first kernel. Just to be clear -- what do you mean by the "second" kernel? Do you mean that a guest kernel crashed guest, and did a kdump operation, and that second kdump kernel failed somehow, and now you're trying to do a "virsh dump" on the kdump kernel? Dave

1 year

3
16
0 / 0

[PATCH makedumpfile 0/2] LZO Compression Support

by HATAYAMA Daisuke

The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering. The RFC discussion: http://lists.infradead.org/pipermail/kexec/2011-November/005783.html http://lists.infradead.org/pipermail/kexec/2011-December/005868.html How to build: 1. Get lzo libraries: lzo, lzo-devel and lzo-minilzo from either of the following: 1) Original author's website: http://www.oberhumer.com/opensource/lzo/ 2) yum framework on fedora. Older releases don't have the packages. 2. Apply the patch set to makedumpfile v1.4.2. 3. Do make. How to use: Introduce new -l option. If a user specify this, makedumpfile generates dumpfile compressed by pages with lzo compression. Example) $ makedumpfile -l vmcore dumpfile Performance evaluation: - Kumagai-san's evaluation simulating actually working servers: http://lists.infradead.org/pipermail/kexec/2011-December/005868.html - My evaluation focusing on the worst cases: http://lists.infradead.org/pipermail/kexec/2011-November/005783.html LZO Support for crash: I'll post LZO support patch for crash after makedumpfile merges these patches. --- HATAYAMA Daisuke (2): Add help and manual messages about LZO compression support Add LZO Support Makefile | 2 +- diskdump_mod.h | 3 ++- makedumpfile.8 | 6 +++--- makedumpfile.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++--------- makedumpfile.h | 2 ++ print_info.c | 16 ++++++++-------- 6 files changed, 64 insertions(+), 22 deletions(-) -- HATAYAMA Daisuke

1 year

2
6
0 / 0

question about phys_base

by Wen Congyang

Hi, Dave I am implementing a new dump command in the qemu. The vmcore's format is elf(like kdump). And I try to provide phys_base in the PT_LOAD. But if the os uses the first vcpu do kdump, the value of phys_base is wrong. I find a function x86_64_virt_phys_base() in crash's code. Is it OK to call this function first? If the function successes, we do not calculate phys_base according to PT_LOAD. Thanks Wen Congyang

1 year

3
6
0 / 0

[PATCH] runq: search current task's runqueue explicitly

by HATAYAMA Daisuke

Currently, runq sub-command doesn't consider CFS runqueue's current task removed from CFS runqueue. Due to this, the remaining CFS runqueus that follow the current task's is not displayed. This patch fixes this by making runq sub-command search current task's runqueue explicitly. Note that CFS runqueue exists for each task group, and so does CFS runqueue's current task, and the above search needs to be done recursively. Test ==== On vmcore I made 7 task groups: root group --- A --- AA --- AAA + +- AAB | +- AB --- ABA +- ABB and then I ran three CPU bound tasks, which is exactly the same as int main(void) { for (;;) continue; return 0; } for each task group, including root group; so total 24 tasks. For readability, I annotated each task name with its belonging group name. For example, loop.ABA belongs to task group ABA. Look at CPU0 collumn below. [before] lacks 8 tasks and [after] successfully shows all tasks on the runqueue, which is identical to the result of [sched debug] that is expected to ouput correct result. I'll send this vmcore later. [before] crash> runq | cat CPU 0 RUNQUEUE: ffff88000a215f80 CURRENT: PID: 28263 TASK: ffff880037aaa040 COMMAND: "loop.ABA" RT PRIO_ARRAY: ffff88000a216098 [no tasks queued] CFS RB_ROOT: ffff88000a216010 [120] PID: 28262 TASK: ffff880037cc40c0 COMMAND: "loop.ABA" <cut> [after] crash_fix> runq CPU 0 RUNQUEUE: ffff88000a215f80 CURRENT: PID: 28263 TASK: ffff880037aaa040 COMMAND: "loop.ABA" RT PRIO_ARRAY: ffff88000a216098 [no tasks queued] CFS RB_ROOT: ffff88000a216010 [120] PID: 28262 TASK: ffff880037cc40c0 COMMAND: "loop.ABA" [120] PID: 28271 TASK: ffff8800787a8b40 COMMAND: "loop.ABB" [120] PID: 28272 TASK: ffff880037afd580 COMMAND: "loop.ABB" [120] PID: 28245 TASK: ffff8800785e8b00 COMMAND: "loop.AB" [120] PID: 28246 TASK: ffff880078628ac0 COMMAND: "loop.AB" [120] PID: 28241 TASK: ffff880078616b40 COMMAND: "loop.AA" [120] PID: 28239 TASK: ffff8800785774c0 COMMAND: "loop.AA" [120] PID: 28240 TASK: ffff880078617580 COMMAND: "loop.AA" [120] PID: 28232 TASK: ffff880079b5d4c0 COMMAND: "loop.A" <cut> [sched debug] crash> runq -d CPU 0 [120] PID: 28232 TASK: ffff880079b5d4c0 COMMAND: "loop.A" [120] PID: 28239 TASK: ffff8800785774c0 COMMAND: "loop.AA" [120] PID: 28240 TASK: ffff880078617580 COMMAND: "loop.AA" [120] PID: 28241 TASK: ffff880078616b40 COMMAND: "loop.AA" [120] PID: 28245 TASK: ffff8800785e8b00 COMMAND: "loop.AB" [120] PID: 28246 TASK: ffff880078628ac0 COMMAND: "loop.AB" [120] PID: 28262 TASK: ffff880037cc40c0 COMMAND: "loop.ABA" [120] PID: 28263 TASK: ffff880037aaa040 COMMAND: "loop.ABA" [120] PID: 28271 TASK: ffff8800787a8b40 COMMAND: "loop.ABB" [120] PID: 28272 TASK: ffff880037afd580 COMMAND: "loop.ABB" <cut> Diff stat ========= defs.h | 1 + task.c | 37 +++++++++++++++++-------------------- 2 files changed, 18 insertions(+), 20 deletions(-) Thanks. HATAYAMA, Daisuke

1 year

2
2
0 / 0

[RFC] makedumpfile, crash: LZO compression support

by HATAYAMA Daisuke

Hello, This is a RFC patch set that adds LZO compression support to makedumpfile and crash utility. LZO is as good as in size but by far better in speed than ZLIB, leading to reducing down time during generation of crash dump and refiltering. How to build: 1. Get LZO library, which is provided as lzo-devel package on recent linux distributions, and is also available on author's website: http://www.oberhumer.com/opensource/lzo/. 2. Apply the patch set to makedumpfile v1.4.0 and crash v6.0.0. 3. Build both using make. But for crash, do the following now: $ make CFLAGS="-llzo2" How to use: I've newly used -l option for lzo compression in this patch. So for example, do as follows: $ makedumpfile -l vmcore dumpfile $ crash vmlinux dumpfile Request of configure-like feature for crash utility: I would like configure-like feature on crash utility for users to select wheather to add LZO feature actually or not in build-time, that is: ./configure --enable-lzo or ./configure --disable-lzo. The reason is that support staff often downloads and installs the latest version of crash utility on machines where lzo library is not provided. Looking at the source code, it looks to me that crash does some kind of configuration processing in a local manner, around configure.c, and I guess it's difficult to use autoconf tools directly. Or is there another better way? Performance Comparison: Sample Data Ideally, I must have measured the performance for many enough vmcores generated from machines that was actually running, but now I don't have enough sample vmcores, I couldn't do so. So this comparison doesn't answer question on I/O time improvement. This is TODO for now. Instead, I choosed worst and best cases regarding compression ratio and speed only. Specifically, the former is /dev/urandom and the latter is /dev/zero. I get the sample data of 10MB, 100MB and 1GB by doing like this: $ dd bs=4096 count=$((1024*1024*1024/4096)) if=/dev/urandom of=urandom.1GB How to measure Then I performed compression for each block, 4096 bytes, and measured total compression time and output size. See attached mycompress.c. Result See attached file result.txt. Discussion For both kinds of data, lzo's compression was considerably quicker than zlib's. Compression ratio is about 37% for urandom data, and about 8.5% for zero data. Actual situation of physical memory would be in between the two cases, and so I guess average compression time ratio is between 37% and 8.5%. Although beyond the topic of this patch set, we can estimate worst compression time on more data size since compression is performed block size wise and the compression time increases linearly. Estimated worst time on 2TB memory is about 15 hours for lzo and about 40 hours for zlib. In this case, compressed data size is larger than the original, so they are really not used, compression time is fully meaningless. I think compression must be done in parallel, and I'll post such patch later. Diffstat * makedumpfile diskdump_mod.h | 3 +- makedumpfile.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++------ makedumpfile.h | 12 +++++++ 3 files changed, 101 insertions(+), 12 deletions(-) * crash defs.h | 1 + diskdump.c | 20 +++++++++++++++++++- diskdump.h | 3 ++- 3 files changed, 22 insertions(+), 2 deletions(-) TODO * evaluation including I/O time using actual vmcores Thanks. HATAYAMA, Daisuke

1 year

3
15
0 / 0

Re: [Crash-utility] [RFI] Support Fujitsu's sadump dump format

by tachibana＠mxm.nes.nec.co.jp

Hi Hatayama-san, On 2011/06/29 12:12:18 +0900, HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com> wrote: > From: Dave Anderson <anderson(a)redhat.com> > Subject: Re: [Crash-utility] [RFI] Support Fujitsu's sadump dump format > Date: Tue, 28 Jun 2011 08:57:42 -0400 (EDT) > > > > > > > ----- Original Message ----- > >> Fujitsu has stand-alone dump mechanism based on firmware level > >> functionality, which we call SADUMP, in short. > >> > >> We've maintained utility tools internally but now we're thinking that > >> the best is crash utility and makedumpfile supports the sadump format > >> for the viewpoint of both portability and maintainability. > >> > >> We'll be of course responsible for its maintainance in a continuous > >> manner. The sadump dump format is very similar to diskdump format and > >> so kdump (compressed) format, so we estimate patch set would be a > >> relatively small size. > >> > >> Could you tell me whether crash utility and makedumpfile can support > >> the sadump format? If OK, we'll start to make patchset. I think it's not bad to support sadump by makedumpfile. However I have several questions. - Do you want to use makedumpfile to make an existing file that sadump has dumped small? - It isn't possible to support the same form as kdump-compressed format now, is it? - When the information that makedumpfile reads from a note of /proc/vmcore (or a header of kdump-compressed format) is added by an extension of makedumpfile, do you need to modify sadump? Thanks tachibana > > > > Sure, yes, the crash utility can always support another dumpfile format. > > > > Thanks. It helps a lot. > > > It's unclear to me how similar SADUMP is to diskdump/compressed-kdump. > > Does your internal version patch diskdump.c, or do you maintain your > > own "sadump.c"? I ask because if your patchset is at all intrusive, > > I'd prefer it be kept in its own file, primarily for maintainability, > > but also because SADUMP is essentially a black-box to anybody outside > > Fujitsu. > > What I meant when I used ``similar'' is both literally and > logically. The format consists of diskdump header-like header, two > kinds of bitmaps used for the same purpose as those in diskump format, > and memory data. They can be handled in common with the existing data > structure, diskdump_data, non-intrusively, so I hope they are placed > in diskdump.c. > > On the other hand, there's a code to be placed at such specific > area. sadump is triggered depending on kdump's progress and so > register values to be contained in vmcore varies according to the > progress: If crash_notes has been initialized when sadump is > triggered, sadump packs the register values in crash_notes; if not > yet, packs registers gathered by firmware. This is sadump specific > processing, so I think putting it in specific sadump.c file is a > natural and reasonable choise. > > Anyway, I have not made any patch set for this. I'll post a patch set > when I complete. > > Again, thanks a lot for the positive answer. > > Thanks. > HATAYAMA, Daisuke > > > _______________________________________________ > kexec mailing list > kexec(a)lists.infradead.org > http://lists.infradead.org/mailman/listinfo/kexec

1 year

3
5
0 / 0

[PATCH makedumpfile v2 0/4] LZO Compression Support

by HATAYAMA Daisuke

The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering. The RFC discussion was made here: http://lists.infradead.org/pipermail/kexec/2011-November/005783.html http://lists.infradead.org/pipermail/kexec/2011-December/005868.html How to build: 1. Get lzo libraries: lzo, lzo-devel and lzo-minilzo from either of the following: 1) Original author's website: http://www.oberhumer.com/opensource/lzo/ 2) yum framework on fedora. Older releases don't have the packages. 2. Apply the patch set to makedumpfile v1.4.2. 3. Do make as follows: $ make USELZO=on Note: In default, no LZO compression support is included. How to use: Introduce new -l option. If a user specify this, makedumpfile generates dumpfile compressed by pages with lzo compression. Example) $ makedumpfile -l vmcore dumpfile Performance evaluation: - Kumagai-san's evaluation simulating working servers: http://lists.infradead.org/pipermail/kexec/2011-December/005868.html - My evaluation focusing on the worst cases: http://lists.infradead.org/pipermail/kexec/2011-November/005783.html LZO Support for crash: I'll post LZO support patch for crash after makedumpfile merges these patches. Changelog: v1 => v2: - Add build condition for LZO support. Enable LZO support if specifying USELZO=on to make command. - Avoid LONG_MAX/ULONG_MAX redefinitions. --- HATAYAMA Daisuke (4): Add build condition for LZO support Add help and manual messages about LZO compression support Avoid LONG_MAX/ULONG_MAX redefinitions Add LZO Support Makefile | 5 ++++ common.h | 4 +++ diskdump_mod.h | 3 ++- makedumpfile.8 | 6 +++-- makedumpfile.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++------ makedumpfile.h | 4 +++ print_info.c | 16 +++++++------ 7 files changed, 86 insertions(+), 19 deletions(-) -- HATAYAMA Daisuke

12 years, 6 months

3
14
0 / 0

[PATCH] add option -E for foreach

by qiaonuohan

Hello Dave, The patch is used to add a new option, "-E regex", to perform the command(s) on all commands whose names contains a match to "regex". "regex" is interpreted as an POSIX extended regular expression. The new option is helpful when using to search some commands whose names are similar. And I think it is more reliable than "foreach <name> <command> | grep xxx", for information comes from command may have a match to "xxx". P.S. The patch uses some function defined in regex.h, which is offered by glibc. -- -- Regards Qiao Nuohan

12 years, 9 months

2
5
0 / 0

[PATCH] display MCNT and PRIVATE when using kmem -p

by Qiao Nuohan

Hello Dave, I was using ‘kmem �p’ to get status of memory. And I could only get "PAGE PHYSICAL MAPPING INDEX CNT FLAGS" in 2.6.x kernel and later, which makes me feel the lack of information. So I think of displaying ‘page._mapcount’ and ‘page.private’, when using ‘kmem -p’. When adding these two items, I found ‘page._count’ is declared to be atomic_t whose definition is: typedef struct { volatile int counter; } atomic_t; However, current crash codes use UINT to get the value of ‘page._count’. The first patch (0001-kmem_p_6.0.2.patch) is used to change UINT to INT, and the second one (0002-kmem_p_6.0.2.patch) will add the items talked above. Both patches are based on crash 6.0.2. BTW, I have tested these two patches on RHEL6.2_x86_64, RHEL6.2_i386, RHEL5.8_x86_64 and RHEL5.8_i386. -- Regards Qiao Nuohan

12 years, 9 months

3
9
0 / 0

question about qemu-generated vmcore

by Wen Congyang

Hi all. As you know, I am implementing a new, dedicated memory dump mechanism. crash cannot get phys_base, because the guest may be in the kdump kernel. As Dave and hatayama suggested, I want to put extra CPU's information in PT_NOTE. For example: $ readelf -n vmcore Notes at offset 0x000001c8 with length 0x00000838: Owner Data size Description CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) QEMU 0x00000557 Unknown note type: (0x00000000) QEMU 0x00000557 Unknown note type: (0x00000000) We put all CPU's information in the QEMU note. But I am not sure which informatin we should put in it. I think the following thing should be put in QEMU note: standard register: rax, rbx, rcx, rdx, rsi, rdi, rsp, rbp, r8, r9, r10, r11, r12, r13, r14, r15 segments: cs, ds, es, fs, gs, ss, ldt, tr, gdt, idt control register: cr0-cr5 other register: rip, rflags Hatayama suggested appending versions at the head of CPU's information. I agree with it, because CPU may have more registers in the future. Any other register or other information should be included in QEMU notes? Any suggestion is welcome. Thanks Wen Congyang

12 years, 9 months

1
0
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Crash-utility February 2012