10:07 p.m.
This patch series add snappy compression support, applied on top of
v1.4.4. snappy is fast compressoin algorhythm like lzo, but it's more
optimized than lzo on x86 machines and some others.
A lot of benchmark is available on the web. Here I only point at my
benchmark I posted a few weeks ago,
- http://lists.infradead.org/pipermail/kexec/2012-June/006425.html
where snappy shows mostly as good performance as simple copying on
the worst case that input data increases its data size during
compression; this means we can use snappy with mostly NO risk.
How to get snappy libraries:
1) Use yum framework to get snappy and snappy-devel packages, or
2) Visit official website: http://code.google.com/p/snappy/,
download snappy-<version>.tar.gz and then build it.
How to build makedumpfile with snappy support:
Do make as follows:
$ make USESNAPPY=on
---
HATAYAMA Daisuke (8):
Add manual description
Add help message
Add uncompression processing
Add compression processing
Notify snappy unsupporting when disabled
Add snappy build support
Add command-line processing for snappy
Add dump header for snappy
Makefile | 5 +++++
diskdump_mod.h | 2 ++
makedumpfile.8 | 9 +++++---
makedumpfile.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++------
makedumpfile.h | 3 +++
print_info.c | 16 ++++++++++-----
6 files changed, 80 insertions(+), 15 deletions(-)
--
Thanks.
HATAYAMA, Daisuke
10:07 p.m.
New subject: [PATCH 1/8] Add dump header for snappy
Signed-off-by: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
---
diskdump_mod.h | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/diskdump_mod.h b/diskdump_mod.h
index e672485..af060b6 100644
--- a/diskdump_mod.h
+++ b/diskdump_mod.h
@@ -80,6 +80,8 @@ struct kdump_sub_header {
/* page flags */
#define DUMP_DH_COMPRESSED_ZLIB 0x1 /* page is compressed with zlib */
#define DUMP_DH_COMPRESSED_LZO 0x2 /* paged is compressed with lzo */
+#define DUMP_DH_COMPRESSED_SNAPPY 0x4
+ /* paged is compressed with snappy */
/* descriptor of each page for vmcore */
typedef struct page_desc {
10:07 p.m.
New subject: [PATCH 2/8] Add command-line processing for snappy
Specify -p option to compress vmcore in snappy.
Signed-off-by: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
---
makedumpfile.c | 5 ++++-
1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/makedumpfile.c b/makedumpfile.c
index d024e95..14fd9f6 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -6968,7 +6968,7 @@ main(int argc, char *argv[])
info->block_order = DEFAULT_ORDER;
message_level = DEFAULT_MSG_LEVEL;
- while ((opt = getopt_long(argc, argv, "b:cDd:EFfg:hi:lMRrsvXx:", longopts,
+ while ((opt = getopt_long(argc, argv, "b:cDd:EFfg:hi:lMpRrsvXx:", longopts,
NULL)) != -1) {
switch (opt) {
case 'b':
@@ -7026,6 +7026,9 @@ main(int argc, char *argv[])
case 'M':
info->flag_dmesg = 1;
break;
+ case 'p':
+ info->flag_compress = DUMP_DH_COMPRESSED_SNAPPY;
+ break;
case 'P':
info->xen_phys_start = strtoul(optarg, NULL, 0);
break;
10:07 p.m.
New subject: [PATCH 3/8] Add snappy build support
Signed-off-by: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
---
Makefile | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)
diff --git a/Makefile b/Makefile
index 7337c58..8e5d72d 100644
--- a/Makefile
+++ b/Makefile
@@ -55,6 +55,11 @@ LIBS := -llzo2 $(LIBS)
CFLAGS += -DUSELZO
endif
+ifeq ($(USESNAPPY), on)
+LIBS := -lsnappy $(LIBS)
+CFLAGS += -DUSESNAPPY
+endif
+
all: makedumpfile
$(OBJ_PART): $(SRC_PART)
10:07 p.m.
New subject: [PATCH 4/8] Notify snappy unsupporting when disabled
Signed-off-by: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
---
makedumpfile.c | 9 +++++++++
1 files changed, 9 insertions(+), 0 deletions(-)
diff --git a/makedumpfile.c b/makedumpfile.c
index 14fd9f6..a47a3ec 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -2538,6 +2538,15 @@ initial(void)
}
#endif
+#ifndef USESNAPPY
+ if (info->flag_compress == DUMP_DH_COMPRESSED_SNAPPY) {
+ MSG("'-p' option is disabled, ");
+ MSG("because this binary doesn't support snappy "
+ "compression.\n");
+ MSG("Try `make USESNAPPY=on` when building.\n");
+ }
+#endif
+
if (!is_xen_memory() && info->flag_exclude_xen_dom) {
MSG("'-X' option is disable,");
MSG("because %s is not Xen's memory core image.\n",
info->name_memory);
10:07 p.m.
New subject: [PATCH 5/8] Add compression processing
Signed-off-by: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
---
makedumpfile.c | 30 +++++++++++++++++++++++++-----
makedumpfile.h | 3 +++
2 files changed, 28 insertions(+), 5 deletions(-)
diff --git a/makedumpfile.c b/makedumpfile.c
index a47a3ec..1525b30 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -4718,6 +4718,7 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data
*cd_page)
struct dump_bitmap bitmap2;
struct timeval tv_start;
const off_t failed = (off_t)-1;
+ unsigned long len_buf_out_zlib, len_buf_out_lzo, len_buf_out_snappy;
int ret = FALSE;
@@ -4726,8 +4727,9 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data
*cd_page)
initialize_2nd_bitmap(&bitmap2);
+ len_buf_out_zlib = len_buf_out_lzo = len_buf_out_snappy = 0;
+
#ifdef USELZO
- unsigned long len_buf_out_zlib, len_buf_out_lzo;
lzo_bytep wrkmem;
if ((wrkmem = malloc(LZO1X_1_MEM_COMPRESS)) == NULL) {
@@ -4736,13 +4738,19 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data
*cd_page)
goto out;
}
- len_buf_out_zlib = compressBound(info->page_size);
len_buf_out_lzo = info->page_size + info->page_size / 16 + 64 + 3;
- len_buf_out = MAX(len_buf_out_zlib, len_buf_out_lzo);
-#else
- len_buf_out = compressBound(info->page_size);
#endif
+#ifdef USESNAPPY
+ len_buf_out_snappy = snappy_max_compressed_length(info->page_size);
+#endif
+
+ len_buf_out_zlib = compressBound(info->page_size);
+
+ len_buf_out = MAX(len_buf_out_zlib,
+ MAX(len_buf_out_lzo,
+ len_buf_out_snappy));
+
if ((buf_out = malloc(len_buf_out)) == NULL) {
ERRMSG("Can't allocate memory for the compression buffer. %s\n",
strerror(errno));
@@ -4843,6 +4851,18 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data
*cd_page)
pd.size = size_out;
memcpy(buf, buf_out, pd.size);
#endif
+#ifdef USESNAPPY
+ } else if ((info->flag_compress & DUMP_DH_COMPRESSED_SNAPPY)
+ && ((size_out = len_buf_out_snappy),
+ snappy_compress((char *)buf, info->page_size,
+ (char *)buf_out,
+ (size_t *)&size_out)
+ == SNAPPY_OK)
+ && (size_out < info->page_size)) {
+ pd.flags = DUMP_DH_COMPRESSED_SNAPPY;
+ pd.size = size_out;
+ memcpy(buf, buf_out, pd.size);
+#endif
} else {
pd.flags = 0;
pd.size = info->page_size;
diff --git a/makedumpfile.h b/makedumpfile.h
index 6f5489d..72d8327 100644
--- a/makedumpfile.h
+++ b/makedumpfile.h
@@ -34,6 +34,9 @@
#ifdef USELZO
#include <lzo/lzo1x.h>
#endif
+#ifdef USESNAPPY
+#include <snappy-c.h>
+#endif
#include "common.h"
#include "dwarf_info.h"
#include "diskdump_mod.h"
10:07 p.m.
New subject: [PATCH 6/8] Add uncompression processing
Signed-off-by: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
---
makedumpfile.c | 16 ++++++++++++++++
1 files changed, 16 insertions(+), 0 deletions(-)
diff --git a/makedumpfile.c b/makedumpfile.c
index 1525b30..e42b07d 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -279,6 +279,22 @@ readpmem_kdump_compressed(unsigned long long paddr, void *bufptr,
size_t size)
}
memcpy(bufptr, buf2 + page_offset, size);
#endif
+#ifdef USESNAPPY
+ } else if ((pd.flags & DUMP_DH_COMPRESSED_SNAPPY)) {
+
+ ret = snappy_uncompressed_length(buf, pd.size, &retlen);
+ if (ret != SNAPPY_OK) {
+ ERRMSG("Uncompress failed: %d\n", ret);
+ goto error;
+ }
+
+ ret = snappy_uncompress(buf, pd.size, buf2, &retlen);
+ if ((ret != SNAPPY_OK) || (retlen != info->page_size)) {
+ ERRMSG("Uncompress failed: %d\n", ret);
+ goto error;
+ }
+ memcpy(bufptr, buf2 + page_offset, size);
+#endif
} else
memcpy(bufptr, buf + page_offset, size);
10:08 p.m.
New subject: [PATCH 7/8] Add help message
Signed-off-by: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
---
print_info.c | 16 +++++++++++-----
1 files changed, 11 insertions(+), 5 deletions(-)
diff --git a/print_info.c b/print_info.c
index 61cafed..1b455b2 100644
--- a/print_info.c
+++ b/print_info.c
@@ -39,6 +39,12 @@ print_usage(void)
#else
MSG(" disabled ('-l' option will be ignored.)\n");
#endif
+ MSG("snappy support:\n");
+#ifdef USESNAPPY
+ MSG(" enabled\n");
+#else
+ MSG(" disabled ('-p' option will be ignored.)\n");
+#endif
MSG("\n");
MSG("Usage:\n");
MSG(" Creating DUMPFILE:\n");
@@ -83,10 +89,10 @@ print_usage(void)
MSG("\n");
MSG("\n");
MSG("Available options:\n");
- MSG(" [-c|-l]:\n");
- MSG(" Compress dump data by each page using zlib for -c option and lzo
for\n");
- MSG(" -l option. A user cannot specify either of these options with -E
option,\n");
- MSG(" because the ELF format does not support compressed data.\n");
+ MSG(" [-c|-l|-p]:\n");
+ MSG(" Compress dump data by each page using zlib for -c option, lzo for -l
option\n");
+ MSG(" or snappy for -p option. A user cannot specify either of these options
with\n");
+ MSG(" -E option, because the ELF format does not support compressed
data.\n");
MSG(" THIS IS ONLY FOR THE CRASH UTILITY.\n");
MSG("\n");
MSG(" [-d DL]:\n");
@@ -222,7 +228,7 @@ print_usage(void)
MSG(" Overwrite DUMPFILE even if it already exists.\n");
MSG("\n");
MSG(" [-h]:\n");
- MSG(" Show help message and LZO support status (enabled/disabled).\n");
+ MSG(" Show help message and LZO/snappy support status
(enabled/disabled).\n");
MSG("\n");
MSG(" [-b <order>]\n");
MSG(" Specify the cache 2^order pages in ram when generating vmcore
info\n");
10:08 p.m.
New subject: [PATCH 8/8] Add manual description
Signed-off-by: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
---
makedumpfile.8 | 9 +++++----
1 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/makedumpfile.8 b/makedumpfile.8
index 1fb9756..e9d4cbe 100644
--- a/makedumpfile.8
+++ b/makedumpfile.8
@@ -121,9 +121,10 @@ configuration, you need to use --diskset option.
.SH OPTIONS
.TP
-\fB\-c,\-l\fR
-Compress dump data by each page using zlib for -c option or lzo for -l option.
-(-l option needs USELZO=on when building.)
+\fB\-c,\-l,\-p\fR
+Compress dump data by each page using zlib for -c option, lzo for -l
+option or snappy for -p option.
+(-l option needs USELZO=on and -p option needs USESNAPPY=on when building)
.br
A user cannot specify this option with \-E option, because the ELF format does
not support compressed data.
@@ -492,7 +493,7 @@ Print debugging message.
.TP
\fB\-h\fR
-Show help message and LZO support status (enabled/disabled).
+Show help message and LZO/snappy support status (enabled/disabled).
.TP
\fB\-v\fR
12:05 a.m.
Hello HATAYAMA-san,
On Tue, 03 Jul 2012 12:07:27 +0900
HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com> wrote:
This patch series add snappy compression support, applied on top of
v1.4.4. snappy is fast compressoin algorhythm like lzo, but it's more
optimized than lzo on x86 machines and some others.
A lot of benchmark is available on the web. Here I only point at my
benchmark I posted a few weeks ago,
- http://lists.infradead.org/pipermail/kexec/2012-June/006425.html
where snappy shows mostly as good performance as simple copying on
the worst case that input data increases its data size during
compression; this means we can use snappy with mostly NO risk.
How to get snappy libraries:
1) Use yum framework to get snappy and snappy-devel packages, or
2) Visit official website: http://code.google.com/p/snappy/,
download snappy-<version>.tar.gz and then build it.
How to build makedumpfile with snappy support:
Do make as follows:
$ make USESNAPPY=on
Thank you for your work.
I will merge your patches into the next version with small addition:
diff --git a/README b/README
index ae986d1..638b111 100644
--- a/README
+++ b/README
@@ -45,6 +45,9 @@
6.Build with lzo support:
# make USELZO=on ; make install
The user has to prepare lzo library.
+ 7.Build with snappy support:
+ # make USESNAPPY=on ; make install
+ The user has to prepare snappy library.
* SUPPORTED KERNELS
This makedumpfile supports the following kernels.
In addition, I did brief performance test with your patches:
- The source data is a vmcore saved on the disk, it might be sparse data.
- makedumpfile writes dumpfile to the same disk as the source data.
- execution time is average of 5 times.
| source | zlib | LZO | snappy
------------------------+---------------+-------------+-------------+-------------
size (byte) | 5,107,498,116 | 242,398,239 | 309,549,499 | 487,542,710
compression ratio(%) | --- | 4.75 | 6.06 | 9.55
execution time (sec) | --- | 143 | 49.6 | 51.2
It seems that LZO still has worth in the case that good compression ratio is expected,
as you said in your benchmark report.
Thanks
Atsushi Kumagai
---
HATAYAMA Daisuke (8):
Add manual description
Add help message
Add uncompression processing
Add compression processing
Notify snappy unsupporting when disabled
Add snappy build support
Add command-line processing for snappy
Add dump header for snappy
Makefile | 5 +++++
diskdump_mod.h | 2 ++
makedumpfile.8 | 9 +++++---
makedumpfile.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++------
makedumpfile.h | 3 +++
print_info.c | 16 ++++++++++-----
6 files changed, 80 insertions(+), 15 deletions(-)
--
Thanks.
HATAYAMA, Daisuke
12:36 a.m.
From: Atsushi Kumagai <kumagai-atsushi(a)mxc.nes.nec.co.jp>
Subject: Re: [PATCH 0/8] makedumpfile: Add Snappy Compression Support
Date: Thu, 5 Jul 2012 14:05:59 +0900
On Tue, 03 Jul 2012 12:07:27 +0900
HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com> wrote:
Thank you for your work.
I will merge your patches into the next version with small addition:
diff --git a/README b/README
index ae986d1..638b111 100644
--- a/README
+++ b/README
@@ -45,6 +45,9 @@
6.Build with lzo support:
# make USELZO=on ; make install
The user has to prepare lzo library.
+ 7.Build with snappy support:
+ # make USESNAPPY=on ; make install
+ The user has to prepare snappy library.
* SUPPORTED KERNELS
This makedumpfile supports the following kernels.
Thanks. I overlooked README file...
In addition, I did brief performance test with your patches:
- The source data is a vmcore saved on the disk, it might be sparse data.
- makedumpfile writes dumpfile to the same disk as the source data.
- execution time is average of 5 times.
| source | zlib | LZO | snappy
------------------------+---------------+-------------+-------------+-------------
size (byte) | 5,107,498,116 | 242,398,239 | 309,549,499 | 487,542,710
compression ratio(%) | --- | 4.75 | 6.06 | 9.55
execution time (sec) | --- | 143 | 49.6 | 51.2
It seems that LZO still has worth in the case that good compression ratio is expected,
as you said in your benchmark report.
Yes, this is just as I'm assuming. I think it's better to use snappy
for the machine with hundreads of gigabytes or more. For the machine
with less than that, it's still reasonable to use LZO because risk at
the worst case is small.
Also, snappy is well optimized for little endian, 64-bit
machines. Though I don't see this actually to be honest, this might
show considerably bad performance on other archs except for x86_64.
Thanks.
HATAYAMA, Daisuke
4527
days inactive
4529
days old
devel@lists.crash-utility.osci.io
10 comments
2 participants
participants (2)
-
Atsushi Kumagai -
HATAYAMA Daisuke