Re: [Crash-utility] [PATCH] Speed up usage of a flat makedumpfile vmcore.

On 04/28/15 17:05, Dave Anderson wrote: > > > ----- Original Message ----- >> Using a bubble sort is slow, switch to an insertion sort. >> >> bubble sort: real 3m45.168s >> >> insertion sort: real 0m3.164s >> >> Signed-off-by: Don Slutz <dslutz(a)verizon.com&gt; >> --- >> I do have a big (32G sized file, that gzipped is 357M). >> let me know if you want it. > > Hi Don, > > I'm running some sanity tests with a couple 94GB flat vmcores by comparing > the data output of some of the more verbose commands. But upon invocation, > I do see some significant speed increases, for example: > > vmcore.1 2m45.545s -> 1m31.428s > vmcore.2 2m42.806s -> 0m46.624s > > The numbers change from invocation to invocation, but always in the > same direction. Probably file page caching is affecting the times. > Yes, file cache size is important here. > Anyway, I've never taken the time to actually understand how the flat > format > is laid out, and I would normally ask the author to sign off on any major > changes to it. But I see that Ken'ichi Ohmichi is no longer a member > of this list, so for all practical purposes, it's currently "unmaintained". > I'll continue testing further tomorrow, but if by chance you can whip up a > simplified explanation of the flat dumpfile layout, I'd appreciate it. > Here is my understanding. The file starts with a header: hyper-0-21-52:~/cores/15.04.27-13:26:11>hexdump -C vmcore.flat | more 00000000 6d 61 6b 65 64 75 6d 70 66 69 6c 65 00 7f 00 00 |makedumpfile....| 00000010 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 |................| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * (from makedumpfile.h): struct makedumpfile_header { char signature[SIG_LEN_MDF]; /* = "makedumpfile" */ int64_t type; int64_t version; }; The 1st data block starts at 4096: 00001000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 40 |...............@| And is 2 64bit integers (from makedumpfile.h): struct makedumpfile_data_header { int64_t offset; int64_t buf_size; }; The offset is the "real file offset" where this data block is. I.E. this is the 1st 0x40 bytes in the normal file. The data header just keeps repeating: 00001050 00 00 00 00 00 00 09 a8 00 00 00 00 00 00 1d 00 |................| 00002d60 00 00 00 00 00 00 26 a8 00 00 00 00 00 01 00 00 |......&.........| 00012d70 00 00 00 00 00 01 26 a8 00 00 00 00 00 01 00 00 |......&.........| Until the offset is -1. Note: The data header is stored in big endian format. While this works for any data type, I have only seen "normal" elf data: 00001010 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 |.ELF............| 00001020 04 00 3e 00 01 00 00 00 00 00 00 00 00 00 00 00 |..>.............| 00001030 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |@...............| 00001040 00 00 00 00 40 00 38 00 2b 00 00 00 00 00 00 00 |....@.8.+.......| 00001060 05 00 00 00 50 01 00 00 01 00 00 00 43 4f 52 45 |....P.......CORE| 00001070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000010e0 00 00 00 00 03 00 00 00 00 00 00 00 e8 0d 33 80 |..............3.| ... So basically what this is a stream way of providing a file so that you can fill out the (for an ELF file) the set of PT_LOAD sections at the end and not need 2 passes (or a local file) to store the result. Hope this helps. -Don Slutz