Hello, This is a RFC patch set that adds LZO compression support to makedumpfile and crash utility. LZO is as good as in size but by far better in speed than ZLIB, leading to reducing down time during generation of crash dump and refiltering. How to build: 1. Get LZO library, which is provided as lzo-devel package on recent linux distributions, and is also available on author's website: http://www.oberhumer.com/opensource/lzo/. 2. Apply the patch set to makedumpfile v1.4.0 and crash v6.0.0. 3. Build both using make. But for crash, do the following now: $ make CFLAGS="-llzo2" How to use: I've newly used -l option for lzo compression in this patch. So for example, do as follows: $ makedumpfile -l vmcore dumpfile $ crash vmlinux dumpfile Request of configure-like feature for crash utility: I would like configure-like feature on crash utility for users to select wheather to add LZO feature actually or not in build-time, that is: ./configure --enable-lzo or ./configure --disable-lzo. The reason is that support staff often downloads and installs the latest version of crash utility on machines where lzo library is not provided.

Looking at the source code, it looks to me that crash does some kind of configuration processing in a local manner, around configure.c, and I guess it's difficult to use autoconf tools directly. Or is there another better way?

----- Original Message ----- > Hello, > > This is a RFC patch set that adds LZO compression support to > makedumpfile and crash utility. LZO is as good as in size but by far > better in speed than ZLIB, leading to reducing down time during > generation of crash dump and refiltering. > > How to build: > > 1. Get LZO library, which is provided as lzo-devel package on recent > linux distributions, and is also available on author's website: > http://www.oberhumer.com/opensource/lzo/. > > 2. Apply the patch set to makedumpfile v1.4.0 and crash v6.0.0. > > 3. Build both using make. But for crash, do the following now: > > $ make CFLAGS="-llzo2" > > How to use: > > I've newly used -l option for lzo compression in this patch. So for > example, do as follows: > > $ makedumpfile -l vmcore dumpfile > $ crash vmlinux dumpfile > > Request of configure-like feature for crash utility: > > I would like configure-like feature on crash utility for users to > select wheather to add LZO feature actually or not in build-time, > that is: ./configure --enable-lzo or ./configure --disable-lzo. > > The reason is that support staff often downloads and installs the > latest version of crash utility on machines where lzo library is not > provided. Right, at least on Fedora, these three packages would need to be installed: lzo-2.06-1.fc15.x86_64 lzo-minilzo-2.06-1.fc15.x86_64 lzo-devel-2.06-1.fc15.x86_64 > > Looking at the source code, it looks to me that crash does some kind > of configuration processing in a local manner, around configure.c, > and I guess it's difficult to use autoconf tools directly. > > Or is there another better way? Instead of using CFLAGS, it would be preferable to dynamically append "-llzo2" to the string that the gdb_merge stanza in the Makefile writes to the "mergelibs" file here: gdb_merge: force @if [ ! -f ${GDB}/README ]; then \ make --no-print-directory gdb_unzip; fi ===> @echo "${LDFLAGS} -lz -ldl -rdynamic" > ${GDB}/gdb/mergelibs @echo "../../${PROGRAM} ../../${PROGRAM}lib.a" > ${GDB}/gdb/mergeobj @if [ ! -f ${GDB}/config.status ]; then \ (cd ${GDB}; ./configure ${GDB_CONF_FLAGS} --with-separate-debug-dir=/usr/lib/debug \ --with-bugurl="" --with-expat=no --with-python=no; \ make --no-print-directory; echo ${TARGET} > crash.target) \ else (cd ${GDB}/gdb; make --no-print-directory;); fi @if [ ! -f ${GDB}/gdb/libgdb.a ]; then \ echo; echo "gdb build failed: ${GDB}/gdb/libgdb.a does not exist"; \ echo; exit 1; fi It also requires an additional #define so that the #include line in defs.h and the calls to lzo_init() and lzo1x_decompress_safe() in diskdump.c can be encapsulated, i.e., something like: #ifdef LZO #include <lzo/lzo1x.h> #endif #ifdef LZO dd->lzo_supported = lzo_init() == LZO_E_OK; #endif #ifdef LZO ret = lzo1x_decompress_safe(dd->compressed_page, pd.size, dd->page_cache_hdr[i].pg_bufptr, &retlen, LZO1X_MEM_DECOMPRESS); #endif The -DLZO would only need to be passed to the diskump.c compile line. This could involve configure.c, but conceivably it could all be done within the Makefile itself. For example, by entering something like "make compress=lzo2", and then adding this to the Makefile: ifneq ($(compress),) ifeq ($(compress), lzo2) LZOFLAG=-DLZO LZOLIB=-llzo2 endif endif And then the diskdump.c compile line could use ${LZOFLAG}, and the gdb_merge stanza could include ${LZOLIB}.

----- Original Message ----- > > This could involve configure.c, but conceivably it could all be > > done within the Makefile itself. For example, by entering something > > like "make compress=lzo2", and then adding this to the Makefile: > > > > ifneq ($(compress),) > > ifeq ($(compress), lzo2) > > LZOFLAG=-DLZO > > LZOLIB=-llzo2 > > endif > > endif > > > > And then the diskdump.c compile line could use ${LZOFLAG}, and the > > gdb_merge stanza could include ${LZOLIB}. > > > > Thanks for accepting the request and for detailed instructions. > > I'll reflect these configuration logic in the next patch. Actually, thinking more about it, I prefer the second solution: > On the other hand, the capability of dynamically adding an optional > library could be a useful feature in general -- not only for LZO > compression support, but for other things in the future. > > I'm thinking that we could allow the user create an "extralibs" file > in the top-level directory, and then build_configure() in configure.c > could read it and dynamically create the LDFLAGS= line in the Makefile. > Currently LDFLAGS can only be modified manually, but the -static example > hasn't worked for a long time, and should be removed. Then the only issue would be the extra -DLZO to be passed to diskdump.c. Since any additional library would pretty much always require a conditional function call or other library-specific reference, a general purpose "extracflags" file could also be created and read by build_configure(), and its contents appended to the Makefile's TARGET_CFLAGS line. With "extralibs" and "extracfiles", we would have a general-purpose solution for adding optional libraries for use by both the top-level crash sources as well as by extension modules that are built within the extensions sub-directory. It would also negate the need to always have to enter special "make ..." command every time -- instead just create the "extralibs" and "extracflags" files, and it gets done automatically.

Performance Comparison: Sample Data Ideally, I must have measured the performance for many enough vmcores generated from machines that was actually running, but now I don't have enough sample vmcores, I couldn't do so. So this comparison doesn't answer question on I/O time improvement. This is TODO for now.

Instead, I choosed worst and best cases regarding compression ratio and speed only. Specifically, the former is /dev/urandom and the latter is /dev/zero. I get the sample data of 10MB, 100MB and 1GB by doing like this: $ dd bs=4096 count=$((1024*1024*1024/4096)) if=/dev/urandom of=urandom.1GB How to measure Then I performed compression for each block, 4096 bytes, and measured total compression time and output size. See attached mycompress.c. Result See attached file result.txt. Discussion For both kinds of data, lzo's compression was considerably quicker than zlib's. Compression ratio is about 37% for urandom data, and about 8.5% for zero data. Actual situation of physical memory would be in between the two cases, and so I guess average compression time ratio is between 37% and 8.5%. Although beyond the topic of this patch set, we can estimate worst compression time on more data size since compression is performed block size wise and the compression time increases linearly. Estimated worst time on 2TB memory is about 15 hours for lzo and about 40 hours for zlib. In this case, compressed data size is larger than the original, so they are really not used, compression time is fully meaningless. I think compression must be done in parallel, and I'll post such patch later. Diffstat * makedumpfile diskdump_mod.h | 3 +- makedumpfile.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++------ makedumpfile.h | 12 +++++++ 3 files changed, 101 insertions(+), 12 deletions(-) * crash defs.h | 1 + diskdump.c | 20 +++++++++++++++++++- diskdump.h | 3 ++- 3 files changed, 22 insertions(+), 2 deletions(-) TODO * evaluation including I/O time using actual vmcores Thanks. HATAYAMA, Daisuke

> Hello Hatayama-san, > > Thank you for your work. > >> Performance Comparison: >> >> Sample Data >> >> Ideally, I must have measured the performance for many enough >> vmcores generated from machines that was actually running, but now >> I don't have enough sample vmcores, I couldn't do so. So this >> comparison doesn't answer question on I/O time improvement. This >> is TODO for now. > > I'll measure the performance for actual vmcores by makedumpfile. > Please wait for a while.

That's very helpful. Thanks in advance. But of course I'm also still looking for alternative way. Thanks. HATAYAMA, Daisuke

> > Hello Hatayama-san, > > > > Thank you for your work. > > > >> Performance Comparison: > >> > >> Sample Data > >> > >> Ideally, I must have measured the performance for many enough > >> vmcores generated from machines that was actually running, but now > >> I don't have enough sample vmcores, I couldn't do so. So this > >> comparison doesn't answer question on I/O time improvement. This > >> is TODO for now. > > > > I'll measure the performance for actual vmcores by makedumpfile. > > Please wait for a while. I measured the performance of makedumpfile for some vmcores. Please see below. Sample Data To simulate a working server, I captured VMCOREs while almost all pages were alloceted and filled with random data. (See attached file "fill_random.c") I captured the VMCOREs of 5GB, 7.5GB and 10GB in the same condition. How to measure I measured the total execution time and the size of output file. $ time makedumpfile --message-level 16 [-c|-l| ] vmcore dumpfile Result See attached file "result.txt". This time, lzo's compression was the quickest, and lzo's compression ratio is almost the same(only a bit worse) as zlib's. It seems good, and I will merge the patch set into the makedumpfile. What is your opinion, Dave? Thanks. KUMAGAI, Atsushi > > That's very helpful. Thanks in advance. > > But of course I'm also still looking for alternative way. > > Thanks. > HATAYAMA, Daisuke >

Hello Hatayama-san, > How about the patch adding compression/IO time report? > I intended it only for the presentation of this RFC. I think the above patch was posted only for this discussion, and I won't merge it into the makedumpfile. > I'll post the corresponding patch in crash's side after crash 6.0.2 is > released, waiting for new configuration editing feature as Dave has explained. If the Crash supports LZO compression, I'll merge the feature into the makedumpfile.

Thanks. KUMAGAI, Atsushi

Hello Kumagai-san, Thanks for the evaluation. So I'll re-post the patch soon removing RFC prefix in header. But there is a remaining fix for checking command-line parameters relevant to addition of the 'l' option. How about the patch adding compression/IO time report? I intended it only for the presentation of this RFC. I'll post the corresponding patch in crash's side after crash 6.0.2 is released, waiting for new configuration editing feature as Dave has explained.

Thanks. HATAYAMA, Daisuke