11:35 p.m.
On 2024/03/12 17:25, Yulong TANG 汤玉龙 wrote:
In Linux 5.1, the ZRAM block driver has changed its default
compressor
from "lzo" to "lzo-rle" to enhance LZO compression support. However,
crash does not support the improved LZO algorithm, resulting in failure
when reading memory.
change default compressor : ce82f19fd5809f0cf87ea9f753c5cc65ca0673d6
The issue was discovered when using the extension 'gcore' to generate a
process coredump, which was found to be incomplete and unable to be
opened properly with gdb.
This patch is for Crash-utility tool, it enables the Crash-utility to
support decompression of the "lzo-rle" compression algorithm used in
zram. The patch has been tested with vmcore files from kernel version
5.4, and successfully allows reading of memory compressed with the zram
compression algorithm.
Testing:
========
before apply this patch :
crash> gcore -v 0 1
gcore: WARNING: only the lzo compressor is supported
gcore: WARNING: only the lzo compressor is supported
gcore: WARNING: only the lzo compressor is supported
after:
crash> gcore -v 0
1 Saved core.1.init
Changelog:
==========
v2: keep the "if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)"
related code of the copied kernel code, but change the "if defined"
macro into a runtime check .
v3: set a default value of HAVE_EFFICIENT_UNALIGNED_ACCESS depending on
architecture, for no ikconfig kernels.
Thanks for the update.
+ } else if (STREQ(name, "lzo-rle")) {
+#ifdef LZO
why are this ifdef and lzo_init() etc. needed? I think we do not use
the lzo library for lzo-rle. maybe I'm missing something..
+ bool efficient_unaligned_access;
There is no need to check this every call, how about making this static?
for example:
static int efficient_unaligned_access = -1;
if (efficient_unaligned_access == -1) {
#if defined(ARM) || defined(ARM64) || defined(X86) || defined(X86_64) ||
defined(PPC) || defined(PPC64) || defined(S390)|| defined(S390X)
efficient_unaligned_access = TRUE;
#else
efficient_unaligned_access = FALSE;
#endif
if ((kt->ikconfig_flags & IKCONFIG_AVAIL) &&
(get_kernel_config("CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS", NULL)
== IKCONFIG_Y)
efficient_unaligned_access = TRUE;
}
Thanks,
Kazu
>
> Patch:
> ==========
>
> See attachment.
>
>
>
> Thanks and regards,
> Yulong
>
>
> --
> Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io
> To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io
> https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
> Contribution Guidelines: https://github.com/crash-utility/crash/wiki
1:38 a.m.
New subject: [PATCH v3 ] Adding the zram decompression algorithm "lzo-rle" to support kernel versions >= 5.1
Hi, Kazu
Thank you for your review.
On 2024/03/12 17:25, Yulong TANG 汤玉龙 wrote:
Thanks for the update.
why are this ifdef and lzo_init() etc. needed? I think we do not use
the lzo library for lzo-rle. maybe I'm missing something..
Yes, you are right. we do not use the lzo library for lzo-rle.
the "lzorle_decompress_safe" function should be directly used here.
There is no need to check this every call, how about making this static?
for example:
static int efficient_unaligned_access = -1;
if (efficient_unaligned_access == -1) {
#if defined(ARM) || defined(ARM64) || defined(X86) || defined(X86_64) ||
defined(PPC) || defined(PPC64) || defined(S390)|| defined(S390X)
efficient_unaligned_access = TRUE;
#else
efficient_unaligned_access = FALSE;
#endif
if ((kt->ikconfig_flags & IKCONFIG_AVAIL) &&
(get_kernel_config("CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS", NULL)
== IKCONFIG_Y)
efficient_unaligned_access = TRUE;
}
This is a good approach to avoid checking every call,
I've made those modifications we talked about, tested, and it's looking good.
Anything else we should optimize?
Thanks,
Yulong
--- a/diskdump.c
+++ b/diskdump.c
@@ -3070,20 +3070,7 @@ try_zram_decompress(ulonglong pte_val, unsigned char *buf, ulong
len, ulonglong
return 0;
#endif
} else if (STREQ(name, "lzo-rle")) {
-#ifdef LZO
- if (!(dd->flags & LZO_SUPPORTED)) {
- if (lzo_init() == LZO_E_OK)
- dd->flags |= LZO_SUPPORTED;
- else
- return 0;
- }
decompressor = (void *)&lzorle_decompress_safe;
-#else
- error(WARNING,
- "zram decompress error: this executable needs to be
built"
- " with lzo-rle library\n");
- return 0;
-#endif
} else { /* todo: support more compressor */
error(WARNING, "only the lzo compressor is supported\n");
return 0;
--- a/lzorle_decompress.c
+++ b/lzorle_decompress.c
@@ -50,16 +50,18 @@ int lzorle_decompress_safe(const unsigned char *in, unsigned long
in_len,
unsigned char bitstream_version;
- bool efficient_unaligned_access;
+ static int efficient_unaligned_access = -1;
+ if (efficient_unaligned_access == -1) {
#if defined(ARM) || defined(ARM64) || defined(X86) || defined(X86_64) || defined(PPC) ||
defined(PPC64) || defined(S390)|| defined(S390X)
- efficient_unaligned_access = true;
+ efficient_unaligned_access = TRUE;
#else
- efficient_unaligned_access = false;
+ efficient_unaligned_access = FALSE;
#endif
- if (kt->ikconfig_flags & IKCONFIG_AVAIL)
- efficient_unaligned_access =
(get_kernel_config("CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS", NULL) ==
IKCONFIG_Y);
+ if ((kt->ikconfig_flags & IKCONFIG_AVAIL) &&
get_kernel_config("CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS", NULL) ==
IKCONFIG_Y)
+ efficient_unaligned_access = TRUE;
+ }
1:38 a.m.
New subject: [PATCH v3 ] Adding the zram decompression algorithm "lzo-rle" to support kernel versions >= 5.1
On 2024/03/26 15:38, Tang Yulong wrote:
Hi, Kazu
Thank you for your review.
> On 2024/03/12 17:25, Yulong TANG 汤玉龙 wrote:
>
> Thanks for the update.
>
>
> why are this ifdef and lzo_init() etc. needed? I think we do not use
> the lzo library for lzo-rle. maybe I'm missing something..
Yes, you are right. we do not use the lzo library for lzo-rle.
the "lzorle_decompress_safe" function should be directly used here.
>
>
> There is no need to check this every call, how about making this static?
> for example:
>
> static int efficient_unaligned_access = -1;
> if (efficient_unaligned_access == -1) {
> #if defined(ARM) || defined(ARM64) || defined(X86) || defined(X86_64) ||
> defined(PPC) || defined(PPC64) || defined(S390)|| defined(S390X)
> efficient_unaligned_access = TRUE;
> #else
> efficient_unaligned_access = FALSE;
> #endif
> if ((kt->ikconfig_flags & IKCONFIG_AVAIL) &&
> (get_kernel_config("CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS", NULL)
> == IKCONFIG_Y)
> efficient_unaligned_access = TRUE;
> }
This is a good approach to avoid checking every call,
I've made those modifications we talked about, tested, and it's looking good.
Anything else we should optimize?
Thank you for testing. a nit, we use 'ulong' for unsigned long usually,
otherwise looks fine to me.
Thanks,
Kazu
>
> Thanks,
> Yulong
>
> --- a/diskdump.c
> +++ b/diskdump.c
> @@ -3070,20 +3070,7 @@ try_zram_decompress(ulonglong pte_val, unsigned char *buf,
ulong len, ulonglong
> return 0;
> #endif
> } else if (STREQ(name, "lzo-rle")) {
> -#ifdef LZO
> - if (!(dd->flags & LZO_SUPPORTED)) {
> - if (lzo_init() == LZO_E_OK)
> - dd->flags |= LZO_SUPPORTED;
> - else
> - return 0;
> - }
> decompressor = (void *)&lzorle_decompress_safe;
> -#else
> - error(WARNING,
> - "zram decompress error: this executable needs to be
built"
> - " with lzo-rle library\n");
> - return 0;
> -#endif
> } else { /* todo: support more compressor */
> error(WARNING, "only the lzo compressor is
supported\n");
> return 0;
>
> --- a/lzorle_decompress.c
> +++ b/lzorle_decompress.c
> @@ -50,16 +50,18 @@ int lzorle_decompress_safe(const unsigned char *in, unsigned long
in_len,
>
> unsigned char bitstream_version;
>
> - bool efficient_unaligned_access;
> + static int efficient_unaligned_access = -1;
>
> + if (efficient_unaligned_access == -1) {
> #if defined(ARM) || defined(ARM64) || defined(X86) || defined(X86_64) ||
defined(PPC) || defined(PPC64) || defined(S390)|| defined(S390X)
> - efficient_unaligned_access = true;
> + efficient_unaligned_access = TRUE;
> #else
> - efficient_unaligned_access = false;
> + efficient_unaligned_access = FALSE;
> #endif
>
> - if (kt->ikconfig_flags & IKCONFIG_AVAIL)
> - efficient_unaligned_access =
(get_kernel_config("CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS", NULL) ==
IKCONFIG_Y);
> + if ((kt->ikconfig_flags & IKCONFIG_AVAIL) &&
get_kernel_config("CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS", NULL) ==
IKCONFIG_Y)
> + efficient_unaligned_access = TRUE;
> + }
>
>
> --
> Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io
> To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io
> https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
> Contribution Guidelines: https://github.com/crash-utility/crash/wiki
1:52 a.m.
New subject: [PATCH v3 ] Adding the zram decompression algorithm "lzo-rle" to support kernel versions >= 5.1
On 2024/03/28 15:38, HAGIO KAZUHITO(萩尾 一仁) wrote:
On 2024/03/26 15:38, Tang Yulong wrote:
> Hi, Kazu
>
> Thank you for your review.
>
>> On 2024/03/12 17:25, Yulong TANG 汤玉龙 wrote:
>>
>> Thanks for the update.
>>
>>
>> why are this ifdef and lzo_init() etc. needed? I think we do not use
>> the lzo library for lzo-rle. maybe I'm missing something..
>
> Yes, you are right. we do not use the lzo library for lzo-rle.
> the "lzorle_decompress_safe" function should be directly used here.
>
>>
>>
>> There is no need to check this every call, how about making this static?
>> for example:
>>
>> static int efficient_unaligned_access = -1;
>> if (efficient_unaligned_access == -1) {
>> #if defined(ARM) || defined(ARM64) || defined(X86) || defined(X86_64) ||
>> defined(PPC) || defined(PPC64) || defined(S390)|| defined(S390X)
>> efficient_unaligned_access = TRUE;
>> #else
>> efficient_unaligned_access = FALSE;
>> #endif
>> if ((kt->ikconfig_flags & IKCONFIG_AVAIL) &&
>> (get_kernel_config("CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS",
NULL)
>> == IKCONFIG_Y)
>> efficient_unaligned_access = TRUE;
>> }
>
> This is a good approach to avoid checking every call,
>
> I've made those modifications we talked about, tested, and it's looking
good.
> Anything else we should optimize?
Thank you for testing. a nit, we use 'ulong' for unsigned long usually,
otherwise looks fine to me.
I found a fatal error during "make extensions":
$ make extensions
gcc -Wall -g -shared -rdynamic -o echo.so echo.c -fPIC -DX86_64 -DLZO
-DSNAPPY -DZSTD -DGDB_10_2
In file included from echo.c:18:
defs.h:54:10: fatal error: lzorle_decompress.h: No such file or directory
#include "lzorle_decompress.h"
^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
...
--- a/defs.h
+++ b/defs.h
@@ -51,6 +51,7 @@
#include <regex.h>
#ifdef LZO
#include <lzo/lzo1x.h>
+#include "lzorle_decompress.h"
#endif
#ifdef SNAPPY
#include <snappy-c.h>
we should move this to diskdump.c ?
Thanks,
Kazu
>
> Thanks,
> Kazu
>
>>
>> Thanks,
>> Yulong
>>
>> --- a/diskdump.c
>> +++ b/diskdump.c
>> @@ -3070,20 +3070,7 @@ try_zram_decompress(ulonglong pte_val, unsigned char *buf,
ulong len, ulonglong
>> return 0;
>> #endif
>> } else if (STREQ(name, "lzo-rle")) {
>> -#ifdef LZO
>> - if (!(dd->flags & LZO_SUPPORTED)) {
>> - if (lzo_init() == LZO_E_OK)
>> - dd->flags |= LZO_SUPPORTED;
>> - else
>> - return 0;
>> - }
>> decompressor = (void *)&lzorle_decompress_safe;
>> -#else
>> - error(WARNING,
>> - "zram decompress error: this executable needs to be
built"
>> - " with lzo-rle library\n");
>> - return 0;
>> -#endif
>> } else { /* todo: support more compressor */
>> error(WARNING, "only the lzo compressor is
supported\n");
>> return 0;
>>
>> --- a/lzorle_decompress.c
>> +++ b/lzorle_decompress.c
>> @@ -50,16 +50,18 @@ int lzorle_decompress_safe(const unsigned char *in, unsigned
long in_len,
>>
>> unsigned char bitstream_version;
>>
>> - bool efficient_unaligned_access;
>> + static int efficient_unaligned_access = -1;
>>
>> + if (efficient_unaligned_access == -1) {
>> #if defined(ARM) || defined(ARM64) || defined(X86) || defined(X86_64) ||
defined(PPC) || defined(PPC64) || defined(S390)|| defined(S390X)
>> - efficient_unaligned_access = true;
>> + efficient_unaligned_access = TRUE;
>> #else
>> - efficient_unaligned_access = false;
>> + efficient_unaligned_access = FALSE;
>> #endif
>>
>> - if (kt->ikconfig_flags & IKCONFIG_AVAIL)
>> - efficient_unaligned_access =
(get_kernel_config("CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS", NULL) ==
IKCONFIG_Y);
>> + if ((kt->ikconfig_flags & IKCONFIG_AVAIL) &&
get_kernel_config("CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS", NULL) ==
IKCONFIG_Y)
>> + efficient_unaligned_access = TRUE;
>> + }
>>
>>
>> --
>> Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io
>> To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io
>> https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
>> Contribution Guidelines: https://github.com/crash-utility/crash/wiki
> --
> Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io
> To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io
> https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
> Contribution Guidelines: https://github.com/crash-utility/crash/wiki
3:20 a.m.
New subject: [PATCH v3 ] Adding the zram decompression algorithm "lzo-rle" to support kernel versions >= 5.1
On 2024/03/28 15:38, HAGIO KAZUHITO(萩尾 一仁) wrote:
I found a fatal error during "make extensions":
$ make extensions
gcc -Wall -g -shared -rdynamic -o echo.so echo.c -fPIC -DX86_64 -DLZO
-DSNAPPY -DZSTD -DGDB_10_2
In file included from echo.c:18:
defs.h:54:10: fatal error: lzorle_decompress.h: No such file or directory
#include "lzorle_decompress.h"
^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
...
--- a/defs.h
+++ b/defs.h
@@ -51,6 +51,7 @@
#include <regex.h>
#ifdef LZO
#include <lzo/lzo1x.h>
+#include "lzorle_decompress.h"
#endif
#ifdef SNAPPY
#include <snappy-c.h>
we should move this to diskdump.c ?
Thanks,
Kazu
Thank you for your review and help,
so sorry for missing that problem.
I have made the modifications we talked about and post v4 patch.
Thanks,
Yulong
> >
> > Thanks,
> > Kazu
> >
> >>
> >> Thanks,
> >> Yulong
> >>
> >> --- a/diskdump.c
> >> +++ b/diskdump.c
> >> @@ -3070,20 +3070,7 @@ try_zram_decompress(ulonglong pte_val, unsigned char
*buf,
> ulong len, ulonglong
> >> return 0;
> >> #endif
> >> } else if (STREQ(name, "lzo-rle")) {
> >> -#ifdef LZO
> >> - if (!(dd->flags & LZO_SUPPORTED)) {
> >> - if (lzo_init() == LZO_E_OK)
> >> - dd->flags |= LZO_SUPPORTED;
> >> - else
> >> - return 0;
> >> - }
> >> decompressor = (void *)&lzorle_decompress_safe;
> >> -#else
> >> - error(WARNING,
> >> - "zram decompress error: this executable needs to
be
> built"
> >> - " with lzo-rle library\n");
> >> - return 0;
> >> -#endif
> >> } else { /* todo: support more compressor */
> >> error(WARNING, "only the lzo compressor is
> supported\n");
> >> return 0;
> >>
> >> --- a/lzorle_decompress.c
> >> +++ b/lzorle_decompress.c
> >> @@ -50,16 +50,18 @@ int lzorle_decompress_safe(const unsigned char *in,
unsigned
> long in_len,
> >>
> >> unsigned char bitstream_version;
> >>
> >> - bool efficient_unaligned_access;
> >> + static int efficient_unaligned_access = -1;
> >>
> >> + if (efficient_unaligned_access == -1) {
> >> #if defined(ARM) || defined(ARM64) || defined(X86) || defined(X86_64) ||
> defined(PPC) || defined(PPC64) || defined(S390)|| defined(S390X)
> >> - efficient_unaligned_access = true;
> >> + efficient_unaligned_access = TRUE;
> >> #else
> >> - efficient_unaligned_access = false;
> >> + efficient_unaligned_access = FALSE;
> >> #endif
> >>
> >> - if (kt->ikconfig_flags & IKCONFIG_AVAIL)
> >> - efficient_unaligned_access =
> (get_kernel_config("CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS", NULL) ==
> IKCONFIG_Y);
> >> + if ((kt->ikconfig_flags & IKCONFIG_AVAIL) &&
> get_kernel_config("CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS", NULL) ==
> IKCONFIG_Y)
> >> + efficient_unaligned_access = TRUE;
> >> + }
> >>
> >>
> >> --
> >> Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io
> >> To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io
> >> https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
> >> Contribution Guidelines: https://github.com/crash-utility/crash/wiki
> > --
> > Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io
> > To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io
> > https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
> > Contribution Guidelines: https://github.com/crash-utility/crash/wiki
242
days inactive
244
days old
devel@lists.crash-utility.osci.io
4 comments
2 participants
participants (2)
-
HAGIO KAZUHITO(萩尾 一仁) -
Tang Yulong