10:36 p.m.
I was asked to look at a s390 SUSE dump which had the Computer Associates
SEOS product modules loaded (seos and ksymadd). They exported symbols
with addresses well below the address space for modules. For example,
"dynamic_Seos_syscall_num" with an address of 0x4f90dc0 which is well
below the base of the first module at 0x7880d000. This results in crash
trying to mmap an anonymous 1.9 GB region. Which naturally fails on a
s390 system where the address space is only 2 GiB in size.
Anyone else run across this? Should crash be able to deal with this or
should I simply tell the customer to stop using CA's SEOS product if they
want us to look at their crash dumps?
--
Kurtis D. Rader, Level 3 Linux Support
ABC Service Center, Linux Change Team
T/L 775-3714, DID +1 503-578-3714
9:32 a.m.
"Kurtis D. Rader" wrote:
I was asked to look at a s390 SUSE dump which had the Computer
Associates
SEOS product modules loaded (seos and ksymadd). They exported symbols
with addresses well below the address space for modules. For example,
"dynamic_Seos_syscall_num" with an address of 0x4f90dc0 which is well
below the base of the first module at 0x7880d000. This results in crash
trying to mmap an anonymous 1.9 GB region. Which naturally fails on a
s390 system where the address space is only 2 GiB in size.
Anyone else run across this? Should crash be able to deal with this or
should I simply tell the customer to stop using CA's SEOS product if they
want us to look at their crash dumps?
Hi Kurt,
I'm deferring this to the IBM guys who deal with s390, SUSE kernels, and
whatever dumpfile format this is (LKCD or s390 native?).
But I'm curious -- where in the crash utliity is an mmap being done?
Dave
--
Kurtis D. Rader, Level 3 Linux Support
ABC Service Center, Linux Change Team
T/L 775-3714, DID +1 503-578-3714
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
9:43 a.m.
Hi Kurt,
This is probably a dumb question, but in your last two posts, you
refer to "crash(1)". In my source code repository, the man page
is crash.8, and in Red Hat kernels, it would be referred to as crash(8).
Is there some other crash variant out there that I'm not aware of?
I presuming the answer is no, but your last reference to an mmap
operation threw me...
Dave
10:45 a.m.
On Wed, 2005-10-26 10:43:58, Dave Anderson wrote:
Hi Kurt,
This is probably a dumb question, but in your last two posts, you refer to
"crash(1)". In my source code repository, the man page is crash.8, and in
Red Hat kernels, it would be referred to as crash(8).
Is there some other crash variant out there that I'm not aware of? I
presuming the answer is no, but your last reference to an mmap operation
threw me...
Sorry for the confusion. For many years I've been in the habit of appending
"(1)" to any command name when used in a sentence where there might be
ambiguity as to whether or not the reference is to a command. I don't
normally bother distinguishing between "normal" and "administrator"
commands in this context.
--
Kurtis D. Rader, Level 3 Linux Support
ABC Service Center, Linux Change Team
T/L 775-3714, DID +1 503-578-3714
10:43 a.m.
On Wed, 2005-10-26 10:32:11, Dave Anderson wrote:
I'm deferring this to the IBM guys who deal with s390, SUSE
kernels, and
whatever dumpfile format this is (LKCD or s390 native?).
Actually I'm pretty confident this is not specific to s390 or SLES. I've
seen the same thing with a dump form a x86 system which had CA SEOS
loaded. It's just that in that case crash is able to create the necessary
memory region because the delta between the addresses is less than 1 GiB
(due, of course to the much smaller size of zone one). So it is still
potentially an issue for x86.
But I'm curious -- where in the crash utliity is an mmap being
done?
From store_module_symbols_v1() -> GETBUF(strbuflen). Where
strbuflen is
calculated as the last module name address minus the first observed
module address plus BUFSIZE. Is this guaranteed to be correct? The code
appears to be assuming the module symbol table is ordered by the address of
the symbol names. Is that guaranteed?
--
Kurtis D. Rader, Level 3 Linux Support
ABC Service Center, Linux Change Team
T/L 775-3714, DID +1 503-578-3714
11:06 a.m.
"Kurtis D. Rader" wrote:
On Wed, 2005-10-26 10:32:11, Dave Anderson wrote:
> I'm deferring this to the IBM guys who deal with s390, SUSE kernels, and
> whatever dumpfile format this is (LKCD or s390 native?).
Actually I'm pretty confident this is not specific to s390 or SLES. I've
seen the same thing with a dump form a x86 system which had CA SEOS
loaded. It's just that in that case crash is able to create the necessary
memory region because the delta between the addresses is less than 1 GiB
(due, of course to the much smaller size of zone one). So it is still
potentially an issue for x86.
> But I'm curious -- where in the crash utliity is an mmap being done?
>From store_module_symbols_v1() -> GETBUF(strbuflen). Where strbuflen is
calculated as the last module name address minus the first observed
module address plus BUFSIZE. Is this guaranteed to be correct? The code
appears to be assuming the module symbol table is ordered by the address of
the symbol names. Is that guaranteed?
Ok, so it's a malloc() failure. Anyway, it's calculating symbol name namespace by
doing
character pointer arithmetic between the pointer to the first symbol's name string
and the last symbol's name string in the module. (The BUFSIZE is fluff.) But
AFAICT,
it shouldn't have anything to do with the value of the symbol, but rather the
pointers
in the module's string section. Sorry but, it's been a while since I looked at
this code,
much less changed it. So the only thing I can possibly think of is that somehow
there's a reference to a module name string that's in another module entirely?
What are the "first" and "last" values, and are they both contained
within the
module's vmalloc'd data area?
Dave
--
Kurtis D. Rader, Level 3 Linux Support
ABC Service Center, Linux Change Team
T/L 775-3714, DID +1 503-578-3714
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
11:20 a.m.
On Wed, 2005-10-26 12:06:20, Dave Anderson wrote:
Ok, so it's a malloc() failure. Anyway, it's calculating
symbol name
namespace by doing character pointer arithmetic between the pointer to the
first symbol's name string and the last symbol's name string in the
module. (The BUFSIZE is fluff.) But AFAICT, it shouldn't have anything
to do with the value of the symbol, but rather the pointers in the
module's string section. Sorry but, it's been a while since I looked at
this code, much less changed it. So the only thing I can possibly think
of is that somehow there's a reference to a module name string that's in
another module entirely? What are the "first" and "last" values, and
are
they both contained within the module's vmalloc'd data area?
That is the problem. The @$^%@#$ CA SEOS module if playing some sort of
game. The first symbol table entry has the name "dynamic_Seos_syscall_num"
with the address 0x4f90dc0. Which is far below the base of the modules at
0x7880d000. There are a few symbols in the CA "ksymadd" module which also
have addresses outside the vmalloc()'ed module space.
This happens on x86 as well. But in that case the delta between the vmalloc
module space addresses and the address of "dynamic_Seos_syscall_num" is only
a few hundred MB, not 1.9 GB.
--
Kurtis D. Rader, Level 3 Linux Support
ABC Service Center, Linux Change Team
T/L 775-3714, DID +1 503-578-3714
8:37 p.m.
New subject: LKCD v9 dumps with non-standard dump header size
The SUSE SLES 9 "bigsmp" kernel configuration defines NR_CPUS == 128. That
causes DUMP_BUFFER_SIZE_MIN to be set to 112 KiB. That is greater than
the default for DUMP_BUFFER_SIZE (which is 64 KiB). Which has the effect
of causing the first page of memory to start 48 KiB farther into the dump
file than crash(1) expects. Which in turn causes crash(1) to croak with a
"conflicting page" error as it tries to handle a bunch of nulls as valid
page data.
The solution is to have crash(1) honor the dh_dump_buffer_size member
of struct __dump_header rather than using the hard-coded constant
LKCD_OFFSET_TO_FIRST_PAGE in lkcd_dump_init_v8(). Of course, that
structure member only exists in version 9 dumps but it didn't seem
worthwhile to create a lkcd_v9.c file just for this one difference.
On the other hand LKCD_OFFSET_TO_FIRST_PAGE is also used in
mclx_cache_page_headers_v8(). The following patch is the minimalist
approach to dealing with this.
--- lkcd_v8.c.old 2005-11-17 03:19:54.246463152 -0800
+++ lkcd_v8.c 2005-11-17 03:22:29.425872296 -0800
@@ -26,6 +26,7 @@ static dump_header_t dump_header_v8 = {
// static dump_header_asm_t dump_header_asm_v8 = { 0 };
static dump_page_t dump_page = { 0 };
static void mclx_cache_page_headers_v8(void);
+static off_t lkcd_offset_to_first_page = LKCD_OFFSET_TO_FIRST_PAGE;
/*
* Verify and initialize the LKCD environment, storing the common data
@@ -146,7 +147,8 @@ lkcd_dump_init_v8(FILE *fp, int fd, char
lkcd->compression = dh->dh_dump_compress;
lkcd->page_header_size = sizeof(dump_page_t);
- lseek(lkcd->fd, LKCD_OFFSET_TO_FIRST_PAGE, SEEK_SET);
+ lkcd_offset_to_first_page = dh_dump_buffer_size;
+ lseek(lkcd->fd, lkcd_offset_to_first_page, SEEK_SET);
/*
* Read all of the pages and save the page offsets for lkcd_lseek().
@@ -483,7 +485,7 @@ mclx_cache_page_headers_v8(void)
/*
* Determine the granularity between offsets.
*/
- if (lseek(lkcd->fd, page_headers[0] + LKCD_OFFSET_TO_FIRST_PAGE,
+ if (lseek(lkcd->fd, page_headers[0] + lkcd_offset_to_first_page,
SEEK_SET) == -1)
return;
if (read(lkcd->fd, dp, lkcd->page_header_size) !=
@@ -491,7 +493,7 @@ mclx_cache_page_headers_v8(void)
return;
physaddr1 = (dp->dp_address - lkcd->kvbase) << lkcd->page_shift;
- if (lseek(lkcd->fd, page_headers[1] + LKCD_OFFSET_TO_FIRST_PAGE,
+ if (lseek(lkcd->fd, page_headers[1] + lkcd_offset_to_first_page,
SEEK_SET) == -1)
return;
if (read(lkcd->fd, dp, lkcd->page_header_size)
@@ -508,7 +510,7 @@ mclx_cache_page_headers_v8(void)
for (i = 0; i < (MCLX_PAGE_HEADERS-1); i++) {
if (!page_headers[i])
break;
- lkcd->curhdroffs = page_headers[i] + LKCD_OFFSET_TO_FIRST_PAGE;
+ lkcd->curhdroffs = page_headers[i] + lkcd_offset_to_first_page;
set_mb_benchmark((granularity * (i+1))/lkcd->page_size);
}
}
--
Kurtis D. Rader, Level 3 Linux Support
ABC Service Center, Linux Change Team
T/L 775-3714, DID +1 503-578-3714
7:46 a.m.
New subject: LKCD v9 dumps with non-standard dump header size
"Kurtis D. Rader" wrote:
The SUSE SLES 9 "bigsmp" kernel configuration defines
NR_CPUS == 128. That
causes DUMP_BUFFER_SIZE_MIN to be set to 112 KiB. That is greater than
the default for DUMP_BUFFER_SIZE (which is 64 KiB). Which has the effect
of causing the first page of memory to start 48 KiB farther into the dump
file than crash(1) expects. Which in turn causes crash(1) to croak with a
"conflicting page" error as it tries to handle a bunch of nulls as valid
page data.
The solution is to have crash(1) honor the dh_dump_buffer_size member
of struct __dump_header rather than using the hard-coded constant
LKCD_OFFSET_TO_FIRST_PAGE in lkcd_dump_init_v8(). Of course, that
structure member only exists in version 9 dumps but it didn't seem
worthwhile to create a lkcd_v9.c file just for this one difference.
On the other hand LKCD_OFFSET_TO_FIRST_PAGE is also used in
mclx_cache_page_headers_v8(). The following patch is the minimalist
approach to dealing with this.
Less is more, I always say...
Looks good to me, unless any other LKCD-ophiles have any objection.
The mclx functions are pretty much cruft these days, since I doubt anybody
on the planet still uses that dump format, but it can't hurt to fix them up too.
Thanks,
Dave
--- lkcd_v8.c.old 2005-11-17 03:19:54.246463152 -0800
+++ lkcd_v8.c 2005-11-17 03:22:29.425872296 -0800
@@ -26,6 +26,7 @@ static dump_header_t dump_header_v8 = {
// static dump_header_asm_t dump_header_asm_v8 = { 0 };
static dump_page_t dump_page = { 0 };
static void mclx_cache_page_headers_v8(void);
+static off_t lkcd_offset_to_first_page = LKCD_OFFSET_TO_FIRST_PAGE;
/*
* Verify and initialize the LKCD environment, storing the common data
@@ -146,7 +147,8 @@ lkcd_dump_init_v8(FILE *fp, int fd, char
lkcd->compression = dh->dh_dump_compress;
lkcd->page_header_size = sizeof(dump_page_t);
- lseek(lkcd->fd, LKCD_OFFSET_TO_FIRST_PAGE, SEEK_SET);
+ lkcd_offset_to_first_page = dh_dump_buffer_size;
+ lseek(lkcd->fd, lkcd_offset_to_first_page, SEEK_SET);
/*
* Read all of the pages and save the page offsets for lkcd_lseek().
@@ -483,7 +485,7 @@ mclx_cache_page_headers_v8(void)
/*
* Determine the granularity between offsets.
*/
- if (lseek(lkcd->fd, page_headers[0] + LKCD_OFFSET_TO_FIRST_PAGE,
+ if (lseek(lkcd->fd, page_headers[0] + lkcd_offset_to_first_page,
SEEK_SET) == -1)
return;
if (read(lkcd->fd, dp, lkcd->page_header_size) !=
@@ -491,7 +493,7 @@ mclx_cache_page_headers_v8(void)
return;
physaddr1 = (dp->dp_address - lkcd->kvbase) << lkcd->page_shift;
- if (lseek(lkcd->fd, page_headers[1] + LKCD_OFFSET_TO_FIRST_PAGE,
+ if (lseek(lkcd->fd, page_headers[1] + lkcd_offset_to_first_page,
SEEK_SET) == -1)
return;
if (read(lkcd->fd, dp, lkcd->page_header_size)
@@ -508,7 +510,7 @@ mclx_cache_page_headers_v8(void)
for (i = 0; i < (MCLX_PAGE_HEADERS-1); i++) {
if (!page_headers[i])
break;
- lkcd->curhdroffs = page_headers[i] + LKCD_OFFSET_TO_FIRST_PAGE;
+ lkcd->curhdroffs = page_headers[i] + lkcd_offset_to_first_page;
set_mb_benchmark((granularity * (i+1))/lkcd->page_size);
}
}
--
Kurtis D. Rader, Level 3 Linux Support
ABC Service Center, Linux Change Team
T/L 775-3714, DID +1 503-578-3714
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
4:14 p.m.
New subject: LKCD v9 dumps with non-standard dump header size
On 11/29/05 08:46, Dave Anderson wrote:
Looks good to me, unless any other LKCD-ophiles have any objection.
We need the following patch to allow crash to still read v8 dumps.
Troy
--- lkcd_v8.c.ori 2005-12-08 14:53:07.026719000 -0700
+++ lkcd_v8.c 2005-12-14 14:52:40.086407560 -0700
@@ -57,10 +57,13 @@
if (read(lkcd->fd, dh, sizeof(dump_header_t)) !=
sizeof(dump_header_t))
return FALSE;
- if ((dh->dh_version & LKCD_DUMP_VERSION_NUMBER_MASK) == LKCD_DUMP_V9)
+ if ((dh->dh_version & LKCD_DUMP_VERSION_NUMBER_MASK) == LKCD_DUMP_V9){
if (read(lkcd->fd, &dh_dump_buffer_size, sizeof(dh_dump_buffer_size)) !=
sizeof(dh_dump_buffer_size))
return FALSE;
+ lkcd_offset_to_first_page = dh_dump_buffer_size;
+ } else
+ lkcd_offset_to_first_page = LKCD_OFFSET_TO_FIRST_PAGE;
lkcd->dump_page = dp;
lkcd->dump_header = dh;
@@ -147,7 +150,6 @@
lkcd->compression = dh->dh_dump_compress;
lkcd->page_header_size = sizeof(dump_page_t);
- lkcd_offset_to_first_page = dh_dump_buffer_size;
lseek(lkcd->fd, lkcd_offset_to_first_page, SEEK_SET);
/*
8:09 a.m.
New subject: LKCD v9 dumps with non-standard dump header size
Troy Heber wrote:
On 11/29/05 08:46, Dave Anderson wrote:
>
> Looks good to me, unless any other LKCD-ophiles have any objection.
We need the following patch to allow crash to still read v8 dumps.
Troy
I don't think that the "else" part is required since
lkcd_offset_to_first_page
is statically initialized to LKCD_OFFSET_TO_FIRST_PAGE above. But
for readability this looks fine to me.
Thanks,
Dave
--- lkcd_v8.c.ori 2005-12-08 14:53:07.026719000 -0700
+++ lkcd_v8.c 2005-12-14 14:52:40.086407560 -0700
@@ -57,10 +57,13 @@
if (read(lkcd->fd, dh, sizeof(dump_header_t)) !=
sizeof(dump_header_t))
return FALSE;
- if ((dh->dh_version & LKCD_DUMP_VERSION_NUMBER_MASK) == LKCD_DUMP_V9)
+ if ((dh->dh_version & LKCD_DUMP_VERSION_NUMBER_MASK) == LKCD_DUMP_V9){
if (read(lkcd->fd, &dh_dump_buffer_size, sizeof(dh_dump_buffer_size))
!=
sizeof(dh_dump_buffer_size))
return FALSE;
+ lkcd_offset_to_first_page = dh_dump_buffer_size;
+ } else
+ lkcd_offset_to_first_page = LKCD_OFFSET_TO_FIRST_PAGE;
lkcd->dump_page = dp;
lkcd->dump_header = dh;
@@ -147,7 +150,6 @@
lkcd->compression = dh->dh_dump_compress;
lkcd->page_header_size = sizeof(dump_page_t);
- lkcd_offset_to_first_page = dh_dump_buffer_size;
lseek(lkcd->fd, lkcd_offset_to_first_page, SEEK_SET);
/*
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
9:45 a.m.
New subject: LKCD v9 dumps with non-standard dump header size
On 12/15/05 09:09, Dave Anderson wrote:
I don't think that the "else" part is required since
lkcd_offset_to_first_page
is statically initialized to LKCD_OFFSET_TO_FIRST_PAGE above. But
for readability this looks fine to me.
It's not the initialization that's the problem, it's the unconditional
assignment of dh_dump_buffer_size in hunk 2 of the patch. That
assignment has to be conditional on the dump version.
Troy
> --- lkcd_v8.c.ori 2005-12-08 14:53:07.026719000 -0700
> +++ lkcd_v8.c 2005-12-14 14:52:40.086407560 -0700
> @@ -57,10 +57,13 @@
> if (read(lkcd->fd, dh, sizeof(dump_header_t)) !=
> sizeof(dump_header_t))
> return FALSE;
> - if ((dh->dh_version & LKCD_DUMP_VERSION_NUMBER_MASK) ==
LKCD_DUMP_V9)
> + if ((dh->dh_version & LKCD_DUMP_VERSION_NUMBER_MASK) ==
LKCD_DUMP_V9){
> if (read(lkcd->fd, &dh_dump_buffer_size,
sizeof(dh_dump_buffer_size)) !=
> sizeof(dh_dump_buffer_size))
> return FALSE;
> + lkcd_offset_to_first_page = dh_dump_buffer_size;
> + } else
> + lkcd_offset_to_first_page = LKCD_OFFSET_TO_FIRST_PAGE;
>
> lkcd->dump_page = dp;
> lkcd->dump_header = dh;
> @@ -147,7 +150,6 @@
> lkcd->compression = dh->dh_dump_compress;
> lkcd->page_header_size = sizeof(dump_page_t);
>
> - lkcd_offset_to_first_page = dh_dump_buffer_size;
> lseek(lkcd->fd, lkcd_offset_to_first_page, SEEK_SET);
>
> /*
9:52 a.m.
New subject: LKCD v9 dumps with non-standard dump header size
On 12/15/05 08:45, Troy Heber wrote:
On 12/15/05 09:09, Dave Anderson wrote:
>
> I don't think that the "else" part is required since
lkcd_offset_to_first_page
> is statically initialized to LKCD_OFFSET_TO_FIRST_PAGE above. But
> for readability this looks fine to me.
>
It's not the initialization that's the problem, it's the unconditional
assignment of dh_dump_buffer_size in hunk 2 of the patch. That
assignment has to be conditional on the dump version.
Doha.. I see what your saying now. Yes, I added the else for
readibility, feel free to nuke it if you want.
Troy
11:08 a.m.
New subject: LKCD v9 dumps with non-standard dump header size
On Thu, 2005-12-15 08:52:33, Troy Heber wrote:
On 12/15/05 08:45, Troy Heber wrote:
>
> It's not the initialization that's the problem, it's the unconditional
> assignment of dh_dump_buffer_size in hunk 2 of the patch. That
> assignment has to be conditional on the dump version.
Doha.. I see what your saying now. Yes, I added the else for
readibility, feel free to nuke it if you want.
Thanks. I created the original patch and was starting to wonder if I knew
how to code :-)
--
Kurtis D. Rader, Level 3 Linux Support
ABC Service Center, Linux Change Team
T/L 775-3714, DID +1 503-578-3714
9:52 a.m.
New subject: LKCD v9 dumps with non-standard dump header size
Troy Heber wrote:
On 12/15/05 09:09, Dave Anderson wrote:
>
> I don't think that the "else" part is required since
lkcd_offset_to_first_page
> is statically initialized to LKCD_OFFSET_TO_FIRST_PAGE above. But
> for readability this looks fine to me.
>
It's not the initialization that's the problem, it's the unconditional
assignment of dh_dump_buffer_size in hunk 2 of the patch. That
assignment has to be conditional on the dump version.
Right -- I understand. All I was saying is this "else" clause:
+ } else
+ lkcd_offset_to_first_page = LKCD_OFFSET_TO_FIRST_PAGE;
should be unnecessary since it defaults to that value anyway?
Whatever... I'm applying your patch as you've posted it.
Thanks,
Dave
Troy
> > --- lkcd_v8.c.ori 2005-12-08 14:53:07.026719000 -0700
> > +++ lkcd_v8.c 2005-12-14 14:52:40.086407560 -0700
> > @@ -57,10 +57,13 @@
> > if (read(lkcd->fd, dh, sizeof(dump_header_t)) !=
> > sizeof(dump_header_t))
> > return FALSE;
> > - if ((dh->dh_version & LKCD_DUMP_VERSION_NUMBER_MASK) ==
LKCD_DUMP_V9)
> > + if ((dh->dh_version & LKCD_DUMP_VERSION_NUMBER_MASK) ==
LKCD_DUMP_V9){
> > if (read(lkcd->fd, &dh_dump_buffer_size,
sizeof(dh_dump_buffer_size)) !=
> > sizeof(dh_dump_buffer_size))
> > return FALSE;
> > + lkcd_offset_to_first_page = dh_dump_buffer_size;
> > + } else
> > + lkcd_offset_to_first_page = LKCD_OFFSET_TO_FIRST_PAGE;
> >
> > lkcd->dump_page = dp;
> > lkcd->dump_header = dh;
> > @@ -147,7 +150,6 @@
> > lkcd->compression = dh->dh_dump_compress;
> > lkcd->page_header_size = sizeof(dump_page_t);
> >
> > - lkcd_offset_to_first_page = dh_dump_buffer_size;
> > lseek(lkcd->fd, lkcd_offset_to_first_page, SEEK_SET);
> >
> > /*
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
6920
days inactive
6970
days old
devel@lists.crash-utility.osci.io
14 comments
3 participants
participants (3)
-
Dave Anderson -
Kurtis D. Rader -
Troy Heber