7:43 p.m.
Hi Vivek,
2007/03/06 12:02:46 +0530, Vivek Goyal <vgoyal(a)in.ibm.com> wrote:
On Thu, Mar 01, 2007 at 09:15:05PM +0900, Ken'ichi Ohmichi wrote:
>
> (added Vivek Goyal)
> (added redhat-kexec-kdump-ml)
>
> Hi everyone,
>
> If Bob's proposal (the end of this mail) is merged into makedumpfile,
> the analysis behavior of ELF dumpfile is different from kdump-compressed
> dumpfile's as follows.
> When reading ELF dumpfile, the crash utility treats the excluded pages
> as zero-filled pages. But, as to kdump-compressed dumpfile, the crash
> utility will display the warning message to mean "These pages are
> excluded by partial dump" when it accesses the excluded pages.
>
I quickly went through this thread. I don't know much about the diskdump
format but from a layman's perspective, instead of keeping the zero pages
and compressing them, why not extend diskdump format and maintain another
bitmap which signifies the valid zero pages but they are not physically
part of the core file? I think overall it might reduce dump size.
Thank you for the comment.
I think that it is unnecessary to extend diskdump format.
Instead of having all the compressed zero-pages, it is enough that
page descriptors of all zero-pages point same zero-page.
In this implement, the size of dumpfile increases by only one page
(4K, 16K, etc.). The attached patch is for it.
Secondly, for ELF format core files, probably we can disable the
behavior of
excluding zero filled pages from dumpfile and then "crash" behavior can
be consistent. (Return zero filled page only for valid data otherwise crib).
I want to use the feature of excluding zero-pages, because our systems
(x86_64) have many zero-pages immediately after system booting.
Bob is researching for the behavior of crash on ELF format dumpfiles.
I would like to wait for his report.
Thanks
Ken'ichi Ohmichi
4:04 p.m.
New subject: handling missing kdump pages in diskdump format
On Wed, 2007-03-07 at 10:43 +0900, Ken'ichi Ohmichi wrote:
I want to use the feature of excluding zero-pages, because our
systems
(x86_64) have many zero-pages immediately after system booting.
Bob is researching for the behavior of crash on ELF format dumpfiles.
I would like to wait for his report.
Sorry you had to wait so long.
Bob Montgomery
Here's what I know about how crash deals with ELF (netdump) dump files
compared to how it deals with kdump (diskdump) dump files.
========================================
Intro to ELF dumpfiles and zero filling:
========================================
ELF format dumpfiles do not contain a page-by-page bitmap of included
and excluded pages. Instead, a program header table describes
groups of contiguous pages that are present in the dumpfile. In its
simplest form, this allows a debugger to locate groups of pages that
are present in the file, and conversely to identify pages that are
missing by failing to find a program header entry that encloses the
address of a missing page.
At some point between the 3.4 and 3.19 versions of crash, code was
added to netdump.c:netdump_read to handle a zero_fill feature in
the ELF files. In a program header entry where p_memsz (MemSiz) is
bigger than p_filsiz (FileSiz), the zone between them is considered
to be filled with zero upon access.
For example, using info from readelf on a -E -d31 dumpfile:
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
...
LOAD 0x00000000004722c8 0xffff810005000000 0x0000000005000000
0x000000000063b000 0x0000000003000000 RWE 0
...
For the group of pages described by this program header entry,
crash-3.19 sets up this internal representation:
{file_offset = 0x4722c8, phys_start = 0x5000000, phys_end = 0x563b000,
zero_fill = 0x8000000}
If the FileSiz and MemSiz are the same, no zero fill zone is needed,
phys_end is the real end of the segment, and zero_fill is set to 0x0.
If the requested address falls between phys_start and phys_end,
it is read from the computed file offset. Otherwise, if it falls
between phys_end and zero_fill, the requested buffer is memset to zero.
Here is an address that falls within the zero fill zone shown above.
(0xffff81000563c000 maps to physical address 0x563c000, which is above
phys_end, but below zero_fill).
crash-3.19> x/xg 0xffff81000563c000
0xffff81000563c000: 0x0000000000000000
gdb (6.4.90-debian) can also read this address from the ELF dumpfile:
(gdb) x/xg 0xffff81000563c000
0xffff81000563c000: 0x0000000000000000
But crash-3.4 did not have the zero fill code and fails:
crash-3.4> x/xg 0xffff81000563c000
0xffff81000563c000: gdb: read error: kernel virtual address:
ffff81000563c000 type: "gdb_readmem_callback"
Cannot access memory at address 0xffff81000563c000
==========================
Philosophical Meandering
==========================
The zero-fill feature gives an ELF dump file *three* ways to represent
pages:
1) Not In The Address Space: There is no program header LOAD entry
that contains the requested address.
2) Not In The File, Zero Fill: A program header LOAD entry contains
the address, but the offset in the segment is between FileSiz
and MemSiz.
3) In The File: A program header LOAD entry contains the address,
and the offset in the segment is smaller than FileSiz.
Unfortunately, makedumpfile recognizes *four* types of pages:
A) Not In The Address Space: physical holes in memory, undumpable zones
B) Excluded Type: Excluded based on page type (user memory, cached page)
C) Zero Content: Excluded based on page content being all 0x0
D) In The File
The problem at hand is that makedumpfile's current mapping of the four
types of pages onto ELF's three types of representation puts both
"B) Excluded Type" and "C) Zero Content" into ELF's "2) Not
In The
File, Zero Fill" representation. This results in crash reporting
the contents of all addresses in excluded pages as 0x0, regardless
of their original value.
We have proposed to fix this problem on diskdump-format dump files by
leaving zero pages in the bitmaps and page descriptors, but pointing
all their data pointers to a single common page of zeroes for access.
Coupled with a modification to crash's diskdump.c:read_diskdump routine
to return SEEK_ERROR on excluded pages instead of zero-filling,
we achieve the goal of reading zeroes only when zeroes were in the
original address. We get an indication of read error when attempting
to access a page that has really been excluded. But we still reduce
the size of the dumpfile by storing one copy of a zero page to serve
as the data image for all zero pages.
======================================================
The Compatibility Concern, and Why It Shouldn't Matter
======================================================
One concern is that if we fix diskdump-format dump files in this way,
they will behave differently than ELF-format dump files. Actually,
they will behave correctly, while ELF-format dump files will still
have the "any excluded page must have contained zeroes" behavior.
So I'd like to assert that even with the old zero-filling behavior,
diskdump files and ELF files did not give the same results. In other
words, you could not count on getting the same result for every
address request, even with crash's zero-filling of excluded pages
working for both cases.
Here is why:
The diskdump format includes a page level bitmap of every page in the
known address space. If makedumpfile wants to exclude a page, it's as
simple as changing a 1 to a 0 in the bitmap and leaving out the page.
A huge expanse of memory with alternating pages excluded wouldn't cause
anything more alarming than a bunch of 0xaaaaaaaa words in the bitmap.
The ELF format requires a program header entry for each distinct
group of pages, with a provision for representing one contiguous
group of excluded pages at the end of the group for later zeroing.
The overhead of creating a program header for every isolated excluded
page would be prohibitive. Because of this, when makedumpfile
builds its map of excluded pages and then translates that into the
ELF format dumpfile, it only acts when it finds groups of 256 or more
contiguous excluded pages. Then it sets up separate program header
entries around the exclusion zone, and continues through the bitmap
until it finds another large contiguous section of excluded pages.
This means that pages that were meant to be excluded, but that were not
in big contiguous groups, get to stay in the dumpfile. And here's an
example, using crash-3.19 on two dumpfiles made from the same vmcore
with an unmodified makedumpfile using -d31, one ELF and one normal
(diskdump format).
=============================================================
Elf (netdump) dumpfile (-E -d31):
=============================================================
crash-3.19> sys
KERNEL: vmlinux-2.6.18-3-telco-amd64
DUMPFILE: dumpfile.E_d31
CPUS: 2
DATE: Tue Feb 6 14:56:05 2007
UPTIME: 00:44:09
LOAD AVERAGE: 0.05, 0.03, 0.05
TASKS: 81
NODENAME: hpde
RELEASE: 2.6.18-3-telco-amd64
VERSION: #1 SMP Mon Feb 5 13:33:27 MST 2007
MACHINE: x86_64 (1800 Mhz)
MEMORY: 3.9 GB
PANIC: "Oops: 0000 [1] SMP " (check log for details)
crash-3.19> x/xg 0xffff810000005ff0
0xffff810000005ff0: 0x0000000000205007
crash-3.19> x/xg 0xffff8100cfe44800
0xffff8100cfe44800: 0xbb67bdc97f9fdefe
=============================================================
kdump (diskdump) dumpfile (-d31):
=============================================================
crash-3.19> sys
KERNEL: ../vmlinux-2.6.18-3-telco-amd64
DUMPFILE: dumpfile.makedumpfile-d31.run1
CPUS: 2
DATE: Tue Feb 6 14:56:05 2007
UPTIME: 00:44:09
LOAD AVERAGE: 0.05, 0.03, 0.05
TASKS: 81
NODENAME: hpde
RELEASE: 2.6.18-3-telco-amd64
VERSION: #1 SMP Mon Feb 5 13:33:27 MST 2007
MACHINE: x86_64 (1800 Mhz)
MEMORY: 3.9 GB
PANIC: "Oops: 0000 [1] SMP " (check log for details)
crash-3.19> x/xg 0xffff810000005ff0
0xffff810000005ff0: 0x0000000000000000
crash-3.19> x/xg 0xffff8100cfe44800
0xffff8100cfe44800: 0x0000000000000000
The two memory accesses that return 0x0000000000000000 in the second
case were not really zero, and the ELF dumpfile doesn't say they were.
So we've never really had address for address compatibility between
an ELF dump and a diskdump dump of the same situation anyway.
With the proposed fixes for the diskdump format, these two accesses
would have correctly given read errors, since the information is not
in the dump file. And that's OK, because the system manager chose
to exclude certain types of pages from the dump. The ELF user could
feel lucky that they accidently got the right answer, only because the
page in question was in a group of 10 contiguous excluded pages instead
of 256 or more, and wasn't really excluded like it should have been.
=====================================
Can ELF Dumpfiles Solve This Problem?
=====================================
To achieve correctness with ELF dumpfiles, one could perhaps remap the
four types of pages to the three types of ELF representations so that
"A) Not In The Address Space" and "B) Excluded Type" were both mapped
to "1) Not In The Address Space". Then "C) Zero Content" would map
to "2) Not In The File, Zero Fill". You would lose the ability to
know if a page were missing because it was never in the address space
in the first place, or because it was excluded because of its type.
But if you read a zero, you'd know it really was a zero.
This method probably means that makedumpfile would have to process a
bitmap of excluded (but not zero) pages against the original program
header table to create a new set of program headers that reflected the
excluded pages, and then go back and process that against a bitmap of
zero pages, to see if any zero-fill tails could be created. That is
left as an exercise for the loyal ELF user.
Otherwise, makedumpfile could quit removing pages because they're
zero, and try to achieve dumpfile size goals by removing pages based
only on type. Then zero pages that were supposed to stay would still
be there, and crash wouldn't need to fill missing pages with zeroes
without knowing whether they really contained zeroes in the first place.
==============
The Last Point
==============
The ELF format is inferior for most serious dumpfile debugging.
It does not support page compression, and its format does not allow
the exclusion of all eligible pages. Whether or not the inferior
ELF format dumpfiles can be made completely correct should not
serve as a barrier to getting it right on diskdump format dumpfiles.
As demonstrated, the two types aren't truly bug for bug compatible now,
so fixing one without fixing the other is still a net win.
10:10 p.m.
New subject: handling missing kdump pages in diskdump format
On Tue, Mar 13, 2007 at 03:04:46PM -0600, Bob Montgomery wrote:
On Wed, 2007-03-07 at 10:43 +0900, Ken'ichi Ohmichi wrote:
> I want to use the feature of excluding zero-pages, because our systems
> (x86_64) have many zero-pages immediately after system booting.
> Bob is researching for the behavior of crash on ELF format dumpfiles.
> I would like to wait for his report.
Sorry you had to wait so long.
Bob Montgomery
Excellent summary Bob.
Here's what I know about how crash deals with ELF (netdump) dump
files
compared to how it deals with kdump (diskdump) dump files.
========================================
Intro to ELF dumpfiles and zero filling:
========================================
ELF format dumpfiles do not contain a page-by-page bitmap of included
and excluded pages. Instead, a program header table describes
groups of contiguous pages that are present in the dumpfile. In its
simplest form, this allows a debugger to locate groups of pages that
are present in the file, and conversely to identify pages that are
missing by failing to find a program header entry that encloses the
address of a missing page.
At some point between the 3.4 and 3.19 versions of crash, code was
added to netdump.c:netdump_read to handle a zero_fill feature in
the ELF files. In a program header entry where p_memsz (MemSiz) is
bigger than p_filsiz (FileSiz), the zone between them is considered
to be filled with zero upon access.
The same is used for handling bss where file image does not occupy space
but program image does. So it makes sense.
[..]
==========================
Philosophical Meandering
==========================
The zero-fill feature gives an ELF dump file *three* ways to represent
pages:
1) Not In The Address Space: There is no program header LOAD entry
that contains the requested address.
2) Not In The File, Zero Fill: A program header LOAD entry contains
the address, but the offset in the segment is between FileSiz
and MemSiz.
3) In The File: A program header LOAD entry contains the address,
and the offset in the segment is smaller than FileSiz.
Unfortunately, makedumpfile recognizes *four* types of pages:
A) Not In The Address Space: physical holes in memory, undumpable zones
B) Excluded Type: Excluded based on page type (user memory, cached page)
C) Zero Content: Excluded based on page content being all 0x0
D) In The File
The problem at hand is that makedumpfile's current mapping of the four
types of pages onto ELF's three types of representation puts both
"B) Excluded Type" and "C) Zero Content" into ELF's "2) Not
In The
File, Zero Fill" representation. This results in crash reporting
the contents of all addresses in excluded pages as 0x0, regardless
of their original value.
I think this is wrong. We should not be mapping excluded pages into "Not
in the File, Zero Fill". I think excluded pages should just not appear
in the program headers at all. Returning zero for an excluded page content
is not right.
=====================================
Can ELF Dumpfiles Solve This Problem?
=====================================
To achieve correctness with ELF dumpfiles, one could perhaps remap the
four types of pages to the three types of ELF representations so that
"A) Not In The Address Space" and "B) Excluded Type" were both
mapped
to "1) Not In The Address Space". Then "C) Zero Content" would map
to "2) Not In The File, Zero Fill". You would lose the ability to
know if a page were missing because it was never in the address space
in the first place, or because it was excluded because of its type.
But if you read a zero, you'd know it really was a zero.
I think this is the way to go. Why would I like to know if a page was
never present or mkdumpfile filtered it out? I think we can live with that
and lets just not create any sort of mapping for excluded pages in finally
generated ELF headers.
Thanks
Vivek
8:34 a.m.
New subject: handling missing kdump pages in diskdump format
Vivek Goyal wrote:
> =====================================
> Can ELF Dumpfiles Solve This Problem?
> =====================================
>
> To achieve correctness with ELF dumpfiles, one could perhaps remap the
> four types of pages to the three types of ELF representations so that
> "A) Not In The Address Space" and "B) Excluded Type" were both
mapped
> to "1) Not In The Address Space". Then "C) Zero Content" would
map
> to "2) Not In The File, Zero Fill". You would lose the ability to
> know if a page were missing because it was never in the address space
> in the first place, or because it was excluded because of its type.
> But if you read a zero, you'd know it really was a zero.
>
I think this is the way to go. Why would I like to know if a page was
never present or mkdumpfile filtered it out? I think we can live with that
and lets just not create any sort of mapping for excluded pages in finally
generated ELF headers.
If "Excluded Type" pages were mapped as "Not in The Address Space",
aren't you going to end up with an absurd number of small PT_LOAD
segments? I believe the original intent was to avoid that.
Dave
1:11 p.m.
New subject: handling missing kdump pages in diskdump format
On Wed, 2007-03-14 at 08:34 -0500, Dave Anderson wrote:
Vivek Goyal wrote:
>
> > =====================================
> > Can ELF Dumpfiles Solve This Problem?
> > =====================================
> >
> > To achieve correctness with ELF dumpfiles, one could perhaps remap the
> > four types of pages to the three types of ELF representations so that
> > "A) Not In The Address Space" and "B) Excluded Type" were
both mapped
> > to "1) Not In The Address Space". Then "C) Zero Content"
would map
> > to "2) Not In The File, Zero Fill". You would lose the ability to
> > know if a page were missing because it was never in the address space
> > in the first place, or because it was excluded because of its type.
> > But if you read a zero, you'd know it really was a zero.
> >
>
> I think this is the way to go. Why would I like to know if a page was
> never present or mkdumpfile filtered it out? I think we can live with that
> and lets just not create any sort of mapping for excluded pages in finally
> generated ELF headers.
>
If "Excluded Type" pages were mapped as "Not in The Address Space",
aren't you going to end up with an absurd number of small PT_LOAD
segments? I believe the original intent was to avoid that.
Dave
This is already covered by the algorithm in the ELF generator in
makedumpfile now, except that the current code lumps the excluded pages
with the zero pages while it scans through the PT_LOAD segments of the
original vmcore. Large contiguous groups of excluded pages would be
identified and removed from the map (probably by turning one PT_LOAD
into two: one on either side of the excluded zone). Smaller isolated
groups of excluded pages that don't meet the group size threshold (worst
case like my example of excluding every odd page in a large zone of
memory) would be ignored by the removal code and simply left in their
existing segments. The threshold value (currently 256 pages) is
adjusted to balance the reduction of the size of the file against the
growth in number of PT_LOAD segments. It doesn't really hurt anything
other than the size of the dump to leave the isolated pages in the dump.
Just like in the example I showed, you can sometimes read a page that
should not have still been there in an ELF dumpfile.
That's why I suggested that the existing makedumpfile code might have to
do two passes to make an ELF dumpfile: One to generate the new longer
list of PT_LOAD segments with the big excluded zones removed, and a
second to see if any large zero areas would allow further changes to the
PT_LOAD segments to put the zero fill areas at the ends and remove the
zero page data images from the dumpfile.
Bob Montgomery
9:18 p.m.
New subject: handling missing kdump pages in diskdump format
Hi Bob,
Thank you for the great report.
2007/03/14 08:40:16 +0530, Vivek Goyal <vgoyal(a)in.ibm.com> wrote:
> =====================================
> Can ELF Dumpfiles Solve This Problem?
> =====================================
>
> To achieve correctness with ELF dumpfiles, one could perhaps remap the
> four types of pages to the three types of ELF representations so that
> "A) Not In The Address Space" and "B) Excluded Type" were both
mapped
> to "1) Not In The Address Space". Then "C) Zero Content" would
map
> to "2) Not In The File, Zero Fill". You would lose the ability to
> know if a page were missing because it was never in the address space
> in the first place, or because it was excluded because of its type.
> But if you read a zero, you'd know it really was a zero.
>
I think this is the way to go. Why would I like to know if a page was
never present or mkdumpfile filtered it out? I think we can live with that
and lets just not create any sort of mapping for excluded pages in finally
generated ELF headers.
In the above way, I worry that the crash utility cannot get the
relocatable information (machdep->machspec->phys_start) on ia64.
At ia64_calc_phys_start(), the crash utility gets phdr->p_paddr
of region 5 memory section as the relocatable information.
If the start of region 5 is excluded, the crash gets different
phdr->p_paddr in the above way.
2007/03/05 16:53:40 -0700, Bob Montgomery <bob.montgomery(a)hp.com> wrote:
> In the above patch, a user cannot distinguish between the
original
> unaccessable pages and the excluded pages.
> I think that the crash utility should display the warning message
> to mean "These pages are excluded by partial dump" when it accesses
> the excluded pages.
I'm not sure how to do this. I think it will require modifying memory.c
in crash to accept another type of return value from read_diskdump,
maybe?
I think that read_diskdump() should display the warning message
before returning SEEK_ERROR.
The following patch is for crash-4.0-3.20:
diff -rpuN crash-4.0-3.20.org/defs.h crash-4.0-3.20/defs.h
--- crash-4.0-3.20.org/defs.h 2007-02-22 05:52:01.000000000 +0900
+++ crash-4.0-3.20/defs.h 2007-03-06 17:06:51.000000000 +0900
@@ -211,6 +211,7 @@ struct number_option {
#define DISKDUMP_LOCAL (0x1)
#define KDUMP_CMPRS_LOCAL (0x2)
+#define NOT_ACCESS_EXCLUDED (0x4)
#define DISKDUMP_VALID() (dd->flags & DISKDUMP_LOCAL)
#define KDUMP_CMPRS_VALID() (dd->flags & KDUMP_CMPRS_LOCAL)
diff -rpuN crash-4.0-3.20.org/diskdump.c crash-4.0-3.20/diskdump.c
--- crash-4.0-3.20.org/diskdump.c 2007-02-22 05:52:01.000000000 +0900
+++ crash-4.0-3.20/diskdump.c 2007-03-06 17:10:11.000000000 +0900
@@ -134,6 +134,8 @@ static int read_dump_header(void)
} else if (!memcmp(header->signature, KDUMP_SIGNATURE,
sizeof(header->signature))) {
dd->flags |= KDUMP_CMPRS_LOCAL;
+ if (header->header_version >= 1)
+ dd->flags |= NOT_ACCESS_EXCLUDED;
} else {
if (CRASHDEBUG(1))
error(INFO, "diskdump: dump does not have panic dump header\n");
@@ -452,6 +454,10 @@ read_diskdump(int fd, void *bufptr, int
if ((pfn >= dd->header->max_mapnr) || !page_is_ram(pfn))
return SEEK_ERROR;
if (!page_is_dumpable(pfn)) {
+ if (dd->flags & NOT_ACCESS_EXCLUDED) {
+ error(INFO, "diskdump: cannot access paddr(%lx) due to the excluded page\n",
paddr);
+ return SEEK_ERROR;
+ }
memset(bufptr, 0, cnt);
return cnt;
}
_
Thanks
Ken'ichi Ohmichi
3:24 p.m.
New subject: handling missing kdump pages in diskdump format
On Tue, 2007-03-20 at 11:18 +0900, Ken'ichi Ohmichi wrote:
Hi Bob,
Thank you for the great report.
You are welcome.
Before continuing the discussion of issues with proposed changes to ELF
dumpfile generation, I'd like to recap where we are and suggest a couple
of actions.
1) Makedumpfile patch: Ken'ichi Ohmichi's email of Wed, 7 Mar 2007
10:43:38 +0900 contained the patch "point_same_zero_page.patch". That
patch contains the nice solution to remove redundant zero page images
from the diskdump dump file by pointing the page descriptors of zero
pages to a common zero image. I suggest that this patch should be
applied to makedumpfile as soon as possible, without waiting on a
possible solution to the ELF situation. As described in my report, ELF
and diskdump dump files have not shown identical behavior in the past.
This patch makes diskdump dump files more accurate, and leaves ELF dump
files at the same level of accuracy that they have always had.
2) Crash patch: Ken'ichi Ohmichi's crash patch in the email of Tue, 20
Mar 2007 11:18:02 +0900 should be submitted to crash as soon as
possible. The error message is a bit wordy, and could be reduced to
prevent wrap on 80 column windows, as suggested below, but that is a
minor issue, since the output is already a bit unwieldy:
Current patch prints on 80-column screen (three levels of crash are now
reporting the problem):
crash> x/xg 0xffff810000005ff0
0xffff810000005ff0: gdb: diskdump: cannot access paddr(5ff0) due to the excl
uded page
gdb: seek error: kernel virtual address: ffff810000005ff0 type: "gdb_readmem_ca
llback"
Cannot access memory at address 0xffff810000005ff0
crash>
With a shortened message in diskdump.c:
crash> x/xg 0xffff810000005ff0
0xffff810000005ff0: gdb: diskdump: paddr(5ff0) excluded from dump
gdb: seek error: kernel virtual address: ffff810000005ff0 type: "gdb_readmem_ca
llback"
Cannot access memory at address 0xffff810000005ff0
crash>
With these two patches, crash reports the contents of diskdump dump
files produced by makedumpfile correctly. Zero content pages that have
not been excluded for other reasons remain accessible, and pages that
have really been excluded become inaccessible, instead of showing 0x0
contents. Crash should continue to read old dump files as before,
because of the change in version number in the dumpfiles.
Bob Montgomery
4:21 p.m.
New subject: handling missing kdump pages in diskdump format
Bob Montgomery wrote:
On Tue, 2007-03-20 at 11:18 +0900, Ken'ichi Ohmichi wrote:
> Hi Bob,
>
> Thank you for the great report.
You are welcome.
Before continuing the discussion of issues with proposed changes to ELF
dumpfile generation, I'd like to recap where we are and suggest a couple
of actions.
1) Makedumpfile patch: Ken'ichi Ohmichi's email of Wed, 7 Mar 2007
10:43:38 +0900 contained the patch "point_same_zero_page.patch". That
patch contains the nice solution to remove redundant zero page images
from the diskdump dump file by pointing the page descriptors of zero
pages to a common zero image. I suggest that this patch should be
applied to makedumpfile as soon as possible, without waiting on a
possible solution to the ELF situation. As described in my report, ELF
and diskdump dump files have not shown identical behavior in the past.
This patch makes diskdump dump files more accurate, and leaves ELF dump
files at the same level of accuracy that they have always had.
2) Crash patch: Ken'ichi Ohmichi's crash patch in the email of Tue, 20
Mar 2007 11:18:02 +0900 should be submitted to crash as soon as
possible. The error message is a bit wordy, and could be reduced to
prevent wrap on 80 column windows, as suggested below, but that is a
minor issue, since the output is already a bit unwieldy:
Current patch prints on 80-column screen (three levels of crash are now
reporting the problem):
When Ken'ichi agrees to and commits the makedumpfile change,
I'll do the same for crash.
BTW, you won't get all the gdb-related clutter if you use the
crash "rd" command to read memory instead of the gdb "x" command.
Anyway, as the patch is currently written, there will be two
error messages, and a "rd" command would look like this:
crash> rd 0xffff810000005ff0
rd: diskdump: paddr(5ff0) excluded from dump
rd: seek error: kernel virtual address: ffff810000005ff0 type: "64-bit KVADDR"
crash>
I guess that's OK. Although I actually kind of liked your
original suggestion for an additional return value to readmem().
That way we could at least consolidate the 2 error messages into
just one.
crash> x/xg 0xffff810000005ff0
0xffff810000005ff0: gdb: diskdump: cannot access paddr(5ff0) due to the excl
uded page
gdb: seek error: kernel virtual address: ffff810000005ff0 type: "gdb_readmem_ca
llback"
Cannot access memory at address 0xffff810000005ff0
crash>
With a shortened message in diskdump.c:
crash> x/xg 0xffff810000005ff0
0xffff810000005ff0: gdb: diskdump: paddr(5ff0) excluded from dump
gdb: seek error: kernel virtual address: ffff810000005ff0 type: "gdb_readmem_ca
llback"
Cannot access memory at address 0xffff810000005ff0
crash>
With these two patches, crash reports the contents of diskdump dump
files produced by makedumpfile correctly. Zero content pages that have
not been excluded for other reasons remain accessible, and pages that
have really been excluded become inaccessible, instead of showing 0x0
contents. Crash should continue to read old dump files as before,
because of the change in version number in the dumpfiles.
I am a little bit worried that this may cause an
unnecessary abort -- based upon your experience
with my suggestion of returning an error instead
of a zero-filled buffer from the current compressed
diskdump format -- so I may be paranoid. Anyway,
I'll probably put in an "out" so that the user
has the choice of getting a zero-filled buffer
like it does now.
Dave
Bob Montgomery
7:26 p.m.
New subject: handling missing kdump pages in diskdump format
Hi,
2007/03/21 16:21:57 -0500, Dave Anderson <anderson(a)redhat.com> wrote:
Bob Montgomery wrote:
> On Tue, 2007-03-20 at 11:18 +0900, Ken'ichi Ohmichi wrote:
> > Hi Bob,
> >
> > Thank you for the great report.
> You are welcome.
>
> Before continuing the discussion of issues with proposed changes to ELF
> dumpfile generation, I'd like to recap where we are and suggest a couple
> of actions.
>
> 1) Makedumpfile patch: Ken'ichi Ohmichi's email of Wed, 7 Mar 2007
> 10:43:38 +0900 contained the patch "point_same_zero_page.patch". That
> patch contains the nice solution to remove redundant zero page images
> from the diskdump dump file by pointing the page descriptors of zero
> pages to a common zero image. I suggest that this patch should be
> applied to makedumpfile as soon as possible, without waiting on a
> possible solution to the ELF situation. As described in my report, ELF
> and diskdump dump files have not shown identical behavior in the past.
> This patch makes diskdump dump files more accurate, and leaves ELF dump
> files at the same level of accuracy that they have always had.
I agree with Bob, I will merge the patch "point_same_zero_page.patch" into
a new makedumpfile. But this change is very important, and I want to check
that this change is correct by doing many tests.
I will release a new makedumpfile until the next weekend.
> With these two patches, crash reports the contents of diskdump
dump
> files produced by makedumpfile correctly. Zero content pages that have
> not been excluded for other reasons remain accessible, and pages that
> have really been excluded become inaccessible, instead of showing 0x0
> contents. Crash should continue to read old dump files as before,
> because of the change in version number in the dumpfiles.
>
I am a little bit worried that this may cause an
unnecessary abort -- based upon your experience
with my suggestion of returning an error instead
of a zero-filled buffer from the current compressed
diskdump format -- so I may be paranoid. Anyway,
I'll probably put in an "out" so that the user
has the choice of getting a zero-filled buffer
like it does now.
I think it is a good idea.
Thanks
Ken'ichi Ohmichi
8:42 p.m.
New subject: handling missing kdump pages in diskdump format
Hi,
2007/03/23 09:26:02 +0900, "Ken'ichi Ohmichi"
<oomichi(a)mxs.nes.nec.co.jp> wrote:
>> 1) Makedumpfile patch: Ken'ichi Ohmichi's email of
Wed, 7 Mar 2007
>> 10:43:38 +0900 contained the patch "point_same_zero_page.patch". That
>> patch contains the nice solution to remove redundant zero page images
>> from the diskdump dump file by pointing the page descriptors of zero
>> pages to a common zero image. I suggest that this patch should be
>> applied to makedumpfile as soon as possible, without waiting on a
>> possible solution to the ELF situation. As described in my report, ELF
>> and diskdump dump files have not shown identical behavior in the past.
>> This patch makes diskdump dump files more accurate, and leaves ELF dump
>> files at the same level of accuracy that they have always had.
I agree with Bob, I will merge the patch "point_same_zero_page.patch" into
a new makedumpfile. But this change is very important, and I want to check
that this change is correct by doing many tests.
I will release a new makedumpfile until the next weekend.
I checked whether this change is correct by the following:
(The following patches are attached with this mail)
- makedumpfile-1.1.2 with "point_same_zero_page2.patch" creates a dumpfile.
- crash-4.0-3.21 with "not-access-excluded-page.patch" analyzes the dumpfile.
- The analysis result of the dumpfile is compared with /proc/vmcore's.
And on i386 linux-2.6.19, I found the difference between the result
of the dumpfile (excluding free pages) and /proc/vmcore's by subcommand
"foreach bt".
But by using crash-4.0-3.21 without "not-access-excluded-page.patch",
there is not any difference. In a word, this difference happens due to
considering the excluded pages as unaccess pages.
It is the diff result of /proc/vmcore's analysis result and the dumpfile's
as follows:
--- result-vmcore.txt 2007-03-28 14:01:20.000000000 +0900
+++ result-dumpfile-d16.txt 2007-03-28 14:01:06.000000000 +0900
@@ -1,7 +1,24 @@
crash> foreach bt
PID: 0 TASK: c037c440 CPU: 0 COMMAND: "swapper"
+bt: diskdump: paddr(44b1ae) excluded from dump
+bt: diskdump: paddr(44b1ae) excluded from dump
+bt: diskdump: paddr(44b2a0) excluded from dump
+bt: diskdump: paddr(44b2a0) excluded from dump
+bt: diskdump: paddr(44b1ae) excluded from dump
+bt: diskdump: paddr(44b1ae) excluded from dump
+bt: diskdump: paddr(44b2a1) excluded from dump
+bt: diskdump: paddr(44b2a1) excluded from dump
+bt: cannot resolve stack trace:
+bt: diskdump: paddr(44b1ae) excluded from dump
+bt: diskdump: paddr(44b1ae) excluded from dump
+bt: diskdump: paddr(44b2a1) excluded from dump
+bt: diskdump: paddr(44b2a1) excluded from dump
#0 [c0446f54] schedule at c0311ef0
- #1 [c0446fcc] cpu_idle at c0102c8d
+bt: text symbols on stack:
+ [c0446fb0] mwait_idle_with_hints at c0102235
+ [c0446fcc] cpu_idle at c0102c92
+ [c0446fd4] start_kernel at c044b56f
+ [c0446fdc] unknown_bootoption at c044b577
PID: 0 TASK: c812f050 CPU: 1 COMMAND: "swapper"
#0 [c8127f3c] schedule at c0311ef0
_
The physical address 0x44b1ae is the symbol start_kernel's.
The text start_kernel is freed at free_initmem() while the kernel
booting, and it is not problem that this text is excluded as free pages.
I did not research the detail of this problem. I guess that the crash
utility expect it can read the text of each process.
I think it is necessary not only the change of handling the excluded
pages but also other changes of the crash utility.
Thanks
Ken'ihci Ohmichi
8:13 a.m.
New subject: handling missing kdump pages in diskdump format
Ken'ichi Ohmichi wrote:
Hi,
2007/03/23 09:26:02 +0900, "Ken'ichi Ohmichi"
<oomichi(a)mxs.nes.nec.co.jp> wrote:
>>> 1) Makedumpfile patch: Ken'ichi Ohmichi's email of Wed, 7 Mar
2007
>>> 10:43:38 +0900 contained the patch "point_same_zero_page.patch".
That
>>> patch contains the nice solution to remove redundant zero page images
>>> from the diskdump dump file by pointing the page descriptors of zero
>>> pages to a common zero image. I suggest that this patch should be
>>> applied to makedumpfile as soon as possible, without waiting on a
>>> possible solution to the ELF situation. As described in my report, ELF
>>> and diskdump dump files have not shown identical behavior in the past.
>>> This patch makes diskdump dump files more accurate, and leaves ELF dump
>>> files at the same level of accuracy that they have always had.
>
>I agree with Bob, I will merge the patch "point_same_zero_page.patch" into
>a new makedumpfile. But this change is very important, and I want to check
>that this change is correct by doing many tests.
>I will release a new makedumpfile until the next weekend.
I checked whether this change is correct by the following:
(The following patches are attached with this mail)
- makedumpfile-1.1.2 with "point_same_zero_page2.patch" creates a dumpfile.
- crash-4.0-3.21 with "not-access-excluded-page.patch" analyzes the dumpfile.
- The analysis result of the dumpfile is compared with /proc/vmcore's.
And on i386 linux-2.6.19, I found the difference between the result
of the dumpfile (excluding free pages) and /proc/vmcore's by subcommand
"foreach bt".
But by using crash-4.0-3.21 without "not-access-excluded-page.patch",
there is not any difference. In a word, this difference happens due to
considering the excluded pages as unaccess pages.
It is the diff result of /proc/vmcore's analysis result and the dumpfile's
as follows:
--- result-vmcore.txt 2007-03-28 14:01:20.000000000 +0900
+++ result-dumpfile-d16.txt 2007-03-28 14:01:06.000000000 +0900
@@ -1,7 +1,24 @@
crash> foreach bt
PID: 0 TASK: c037c440 CPU: 0 COMMAND: "swapper"
+bt: diskdump: paddr(44b1ae) excluded from dump
+bt: diskdump: paddr(44b1ae) excluded from dump
+bt: diskdump: paddr(44b2a0) excluded from dump
+bt: diskdump: paddr(44b2a0) excluded from dump
+bt: diskdump: paddr(44b1ae) excluded from dump
+bt: diskdump: paddr(44b1ae) excluded from dump
+bt: diskdump: paddr(44b2a1) excluded from dump
+bt: diskdump: paddr(44b2a1) excluded from dump
+bt: cannot resolve stack trace:
+bt: diskdump: paddr(44b1ae) excluded from dump
+bt: diskdump: paddr(44b1ae) excluded from dump
+bt: diskdump: paddr(44b2a1) excluded from dump
+bt: diskdump: paddr(44b2a1) excluded from dump
#0 [c0446f54] schedule at c0311ef0
- #1 [c0446fcc] cpu_idle at c0102c8d
+bt: text symbols on stack:
+ [c0446fb0] mwait_idle_with_hints at c0102235
+ [c0446fcc] cpu_idle at c0102c92
+ [c0446fd4] start_kernel at c044b56f
+ [c0446fdc] unknown_bootoption at c044b577
PID: 0 TASK: c812f050 CPU: 1 COMMAND: "swapper"
#0 [c8127f3c] schedule at c0311ef0
_
The physical address 0x44b1ae is the symbol start_kernel's.
The text start_kernel is freed at free_initmem() while the kernel
booting, and it is not problem that this text is excluded as free pages.
I did not research the detail of this problem. I guess that the crash
utility expect it can read the text of each process.
The x86 backtracer is constantly reading text to determine the size
of a stack frame from a given return address. However, there are a
number of traps in the backtrace code, however, to recognize "cpu_idle"
as a backtrace ending point, but it still must be reading back farther.
I think it is necessary not only the change of handling the excluded
pages but also other changes of the crash utility.
Can you make the vmlinux/vmcore pair available to me? Then I
fix this particular issue as well as testing my implementation of the
crash utility's excluded page handling.
Thanks,
Dave
8:31 a.m.
New subject: handling missing kdump pages in diskdump format
Dave Anderson wrote:
Can you make the vmlinux/vmcore pair available to me? Then I
fix this particular issue as well as testing my implementation of the
crash utility's excluded page handling.
Thanks,
Dave
On another note, would it be possible to pass the "dump_level" in the
diskdump header somehow -- while maintaining backwards
compatibility with compressed-diskdump files? Perhaps in the
upper bits of the diskdump->header_version, or something
like that?
I think it would be useful to know exactly which types of pages
were excluded from a dumpfile when analyzing it.
Dave
3:52 a.m.
New subject: handling missing kdump pages in diskdump format
Hi Dave,
2007/03/29 08:31:54 -0500, Dave Anderson <anderson(a)redhat.com> wrote:
Dave Anderson wrote:
>
> Can you make the vmlinux/vmcore pair available to me? Then I
> fix this particular issue as well as testing my implementation of the
> crash utility's excluded page handling.
>
> Thanks,
> Dave
On another note, would it be possible to pass the "dump_level" in the
diskdump header somehow -- while maintaining backwards
compatibility with compressed-diskdump files? Perhaps in the
upper bits of the diskdump->header_version, or something
like that?
I think it would be useful to know exactly which types of pages
were excluded from a dumpfile when analyzing it.
I think it is good that the crash utility can know the dump_level
of a dumpfile. But it is not good that dump_level is merged into
diskdump->header_version, because one member should represent one
meaning. I checked the code of crash-4.0-3.21, and I noticed
diskdump->total_ram_blocks is not referred by the crash utility
though the diskdump module sets the value into it. And makedumpfile
sets 0 into it. It does not have any meaning.
Can this member be changed for dump_level ?
I think the following methods for it:
- If the crash utility reads a dumpfile, it checks diskdump->
header_version.
- If diskdump->header_version is 1 or more, the crash utility
considers diskdump->total_ram_blocks as dump_level.
By the way, the dump_level of diskdump is different from the one
of kdump (makedumpfile). If the crash utility will be able to
display the excluded page-type, we should note it.
The dump_level of diskdump
1: Excluding cache pages with private pages
2: Excluding zero-filled pages
4: Excluding free pages
8: Excluding user process data pages
16: Saving private pages
The dump_level of kdump (makedumpfile)
1: Excluding zero-filled pages
2: Excluding cache pages without private pages
4: Excluding cache pages with private pages
8: Excluding user process data pages
16: Excluding free pages
Thanks
Ken'ichi Ohmichi
7:54 a.m.
New subject: handling missing kdump pages in diskdump format
Ken'ichi Ohmichi wrote:
Hi Dave,
2007/03/29 08:31:54 -0500, Dave Anderson <anderson(a)redhat.com> wrote:
>Dave Anderson wrote:
>
>>
>> Can you make the vmlinux/vmcore pair available to me? Then I
>> fix this particular issue as well as testing my implementation of the
>> crash utility's excluded page handling.
>>
>> Thanks,
>> Dave
>
>On another note, would it be possible to pass the "dump_level" in the
>diskdump header somehow -- while maintaining backwards
>compatibility with compressed-diskdump files? Perhaps in the
>upper bits of the diskdump->header_version, or something
>like that?
>
>I think it would be useful to know exactly which types of pages
>were excluded from a dumpfile when analyzing it.
I think it is good that the crash utility can know the dump_level
of a dumpfile. But it is not good that dump_level is merged into
diskdump->header_version, because one member should represent one
meaning. I checked the code of crash-4.0-3.21, and I noticed
diskdump->total_ram_blocks is not referred by the crash utility
though the diskdump module sets the value into it. And makedumpfile
sets 0 into it. It does not have any meaning.
Can this member be changed for dump_level ?
I think the following methods for it:
- If the crash utility reads a dumpfile, it checks diskdump->
header_version.
- If diskdump->header_version is 1 or more, the crash utility
considers diskdump->total_ram_blocks as dump_level.
By the way, the dump_level of diskdump is different from the one
of kdump (makedumpfile). If the crash utility will be able to
display the excluded page-type, we should note it.
The dump_level of diskdump
1: Excluding cache pages with private pages
2: Excluding zero-filled pages
4: Excluding free pages
8: Excluding user process data pages
16: Saving private pages
The dump_level of kdump (makedumpfile)
1: Excluding zero-filled pages
2: Excluding cache pages without private pages
4: Excluding cache pages with private pages
8: Excluding user process data pages
16: Excluding free pages
Thanks
Ken'ichi Ohmichi
Sounds good to me. I don't have any plans for the translation
of the dump_level, but only a display of its value in a new "help -n"
function that I've written for diskdump dumpfiles. "help -n" shows
netdump, ELF-diskdump, kdump, LKCD and xendump dumpfile
information, but functions for diskdump and s390 dumps were
never written.
Thanks,
Dave
6:03 a.m.
New subject: handling missing kdump pages in diskdump format
Hi Dave,
2007/04/02 08:54:38 -0400, Dave Anderson <anderson(a)redhat.com> wrote:
> I think it is good that the crash utility can know the
dump_level
> of a dumpfile. But it is not good that dump_level is merged into
> diskdump->header_version, because one member should represent one
> meaning. I checked the code of crash-4.0-3.21, and I noticed
> diskdump->total_ram_blocks is not referred by the crash utility
> though the diskdump module sets the value into it. And makedumpfile
> sets 0 into it. It does not have any meaning.
>
> Can this member be changed for dump_level ?
> I think the following methods for it:
> - If the crash utility reads a dumpfile, it checks diskdump->
> header_version.
> - If diskdump->header_version is 1 or more, the crash utility
> considers diskdump->total_ram_blocks as dump_level.
>
> By the way, the dump_level of diskdump is different from the one
> of kdump (makedumpfile). If the crash utility will be able to
> display the excluded page-type, we should note it.
>
> The dump_level of diskdump
> 1: Excluding cache pages with private pages
> 2: Excluding zero-filled pages
> 4: Excluding free pages
> 8: Excluding user process data pages
> 16: Saving private pages
>
> The dump_level of kdump (makedumpfile)
> 1: Excluding zero-filled pages
> 2: Excluding cache pages without private pages
> 4: Excluding cache pages with private pages
> 8: Excluding user process data pages
> 16: Excluding free pages
>
> Thanks
> Ken'ichi Ohmichi
Sounds good to me. I don't have any plans for the translation
of the dump_level, but only a display of its value in a new "help -n"
function that I've written for diskdump dumpfiles. "help -n" shows
netdump, ELF-diskdump, kdump, LKCD and xendump dumpfile
information, but functions for diskdump and s390 dumps were
never written.
It is a good idea that "help -n" shows dump_level.
I checked the code of diskdumputils-1.3.25, and I found the dumpfilter
command refers to diskdump->total_ram_blocks. In struct disk_dump_header,
there is no member not referred to by neither the crash utility or diskdumputils.
My proposal (changing total_ram_blocks for dump_level) was not good.
I think it is enough that the member for dump_level is added into the
sub header of kdump (struct kdump_sub_header) only for kdump.
Is the change for diskdump necessary, too ?
Thanks
Ken'ichi Ohmichi
8:40 a.m.
New subject: handling missing kdump pages in diskdump format
Ken'ichi Ohmichi wrote:
Hi Dave,
2007/04/02 08:54:38 -0400, Dave Anderson <anderson(a)redhat.com> wrote:
>> I think it is good that the crash utility can know the dump_level
>> of a dumpfile. But it is not good that dump_level is merged into
>> diskdump->header_version, because one member should represent one
>> meaning. I checked the code of crash-4.0-3.21, and I noticed
>> diskdump->total_ram_blocks is not referred by the crash utility
>> though the diskdump module sets the value into it. And makedumpfile
>> sets 0 into it. It does not have any meaning.
>>
>> Can this member be changed for dump_level ?
>> I think the following methods for it:
>> - If the crash utility reads a dumpfile, it checks diskdump->
>> header_version.
>> - If diskdump->header_version is 1 or more, the crash utility
>> considers diskdump->total_ram_blocks as dump_level.
>>
>> By the way, the dump_level of diskdump is different from the one
>> of kdump (makedumpfile). If the crash utility will be able to
>> display the excluded page-type, we should note it.
>>
>> The dump_level of diskdump
>> 1: Excluding cache pages with private pages
>> 2: Excluding zero-filled pages
>> 4: Excluding free pages
>> 8: Excluding user process data pages
>> 16: Saving private pages
>>
>> The dump_level of kdump (makedumpfile)
>> 1: Excluding zero-filled pages
>> 2: Excluding cache pages without private pages
>> 4: Excluding cache pages with private pages
>> 8: Excluding user process data pages
>> 16: Excluding free pages
>>
>> Thanks
>> Ken'ichi Ohmichi
>
>Sounds good to me. I don't have any plans for the translation
>of the dump_level, but only a display of its value in a new "help -n"
>function that I've written for diskdump dumpfiles. "help -n" shows
>netdump, ELF-diskdump, kdump, LKCD and xendump dumpfile
>information, but functions for diskdump and s390 dumps were
>never written.
It is a good idea that "help -n" shows dump_level.
I checked the code of diskdumputils-1.3.25, and I found the dumpfilter
command refers to diskdump->total_ram_blocks. In struct disk_dump_header,
there is no member not referred to by neither the crash utility or diskdumputils.
My proposal (changing total_ram_blocks for dump_level) was not good.
I think it is enough that the member for dump_level is added into the
sub header of kdump (struct kdump_sub_header) only for kdump.
Is the change for diskdump necessary, too ?
If this were to be restricted to kdump only, then it would be
safe to add a field to the kdump_sub_header since it contains
only one field.
Whether it is necessary for diskdump is up to the diskdump
maintainers to decide. But if they want to do it, they would
have to come up with different manner of adding it to the
disk_dump_sub_header structure, because of the way
that it's declared, i.e., using "long elf_regs" as a placeholder
to describe the start of the processor-dependent register
set. They'd need to define a new disk_dump_sub_header
version that alternatively would be used depending upon
the header->version number.
Or again, since header->version has plenty of space,
I still think it makes just as much sense to utilize the
upper bits for something useful:
But it is not good that dump_level is merged into
diskdump->header_version, because one member should represent one
meaning
I agree with the general statement, but until now, the version
field been completely useless, without any "meaning".
But that's just my opinion, and I understand your hesitation at
doing it that way. Just let me know what you're planning to do...
Thanks,
Dave
3:41 p.m.
New subject: handling missing kdump pages in diskdump format
Dave Anderson wrote:
Actually, upon looking at a sample ELF-format dumpfile from
Ken'ichi,
and hacking in a forced-readmem()-failure upon accessing the
page containing the start_kernel __init function, I can reproduce
the problem with the 2.6.19 kernel. But it is not a problem with
makedumpfile's excluded pages, but rather with the backtrace code's
framesize calculation of schedule() -- which is causing it skip over
the cpu_idle() ending point. It seems to be specific to 2.6.19 for
some reason. I'll look further into that issue.
I've tracked down why the framesize calculation of schedule() is
too large in this case, and therefore skipping over the calling
function's (cpu_idle) frame, and going too far back up the stack
into the "start_kernel" frame reference left on the stack.
It's because there is a BUG() instruction in schedule() code,
which has the line number and virtual address of the filename
string encoded just after the BUG()'s "ud2a" instruction.
#define BUG() \
__asm__ __volatile__( "ud2\n" \
"\t.word %c0\n" \
"\t.long %c1\n" \
: : "i" (__LINE__), "i" (__FILE__))
It just so happens that the virtual address of the filename,
when erroneously decoded as i386 instructions, by dumb luck
translates into a "pusha" instruction ("push all general-purpose
registers"), and therefore the framesize calculator erroneously
adds 32 bytes.
It's weird (and fortunate), that I've spent the last couple of
days coming up with a "dis" fix to deal with those extra bytes
encoded after x86 and x86_64 "ud2a" instructions that causes the
text disassembly to go off into the weeds. Anyway, I'll leverage
off of that code to make the framesize calculator also skip over
the extra encoded bytes instead of translating them.
This could very well be the cause of other x86 framesize
miscalculations that has led to incorrect backtraces. Only
because Bob had put in the code to refuse excluded-page
read attempts was this actually caught in the act.
Thanks, Bob!
Dave
11 p.m.
New subject: handling missing kdump pages in diskdump format
Hi Dave,
2007/04/03 09:40:56 -0400, Dave Anderson <anderson(a)redhat.com> wrote:
> It is a good idea that "help -n" shows dump_level.
>
> I checked the code of diskdumputils-1.3.25, and I found the dumpfilter
> command refers to diskdump->total_ram_blocks. In struct disk_dump_header,
> there is no member not referred to by neither the crash utility or diskdumputils.
> My proposal (changing total_ram_blocks for dump_level) was not good.
>
> I think it is enough that the member for dump_level is added into the
> sub header of kdump (struct kdump_sub_header) only for kdump.
> Is the change for diskdump necessary, too ?
>
>
If this were to be restricted to kdump only, then it would be
safe to add a field to the kdump_sub_header since it contains
only one field.
Whether it is necessary for diskdump is up to the diskdump
maintainers to decide. But if they want to do it, they would
have to come up with different manner of adding it to the
disk_dump_sub_header structure, because of the way
that it's declared, i.e., using "long elf_regs" as a placeholder
to describe the start of the processor-dependent register
set. They'd need to define a new disk_dump_sub_header
version that alternatively would be used depending upon
the header->version number.
We asked the diskdump developers (Indoh-san, Akiyama-san) how
to display diskdump's dump_level, and they said that changing
of file format is not necessary for displaying the dump_level.
And they gave us a good advice that the crash utility can get
diskdump's dump_level by using a symbol "dump_level" like the
following:
int get_dump_level(void)
{
if (symbol_exists("dump_level")) {
int dump_level;
readmem(symbol_value("dump_level"), KVADDR, &dump_level,
sizeof(dump_level), "dump_level",
FAULT_ON_ERROR);
return dump_level;
}
else
...
I think it is a good way because the crash utility can get diskdump's
dump_level of the existing dumpfile. I propose the following way for it.
Please let me know your opinion.
- If a symbol "dump_level" is present (diskdump), the dump_level is
taken from a symbol "dump_level".
- Else if a dumpfile is kdump's and header_version is 1 or more, the
dump_level taken from a new member "kdump_sub_header->dump_level".
Thanks
Ken'ichi Ohmichi
8:04 a.m.
New subject: handling missing kdump pages in diskdump format
Ken'ichi Ohmichi wrote:
Hi Dave,
2007/04/03 09:40:56 -0400, Dave Anderson <anderson(a)redhat.com> wrote:
>> It is a good idea that "help -n" shows dump_level.
>>
>> I checked the code of diskdumputils-1.3.25, and I found the dumpfilter
>> command refers to diskdump->total_ram_blocks. In struct disk_dump_header,
>> there is no member not referred to by neither the crash utility or
diskdumputils.
>> My proposal (changing total_ram_blocks for dump_level) was not good.
>>
>> I think it is enough that the member for dump_level is added into the
>> sub header of kdump (struct kdump_sub_header) only for kdump.
>> Is the change for diskdump necessary, too ?
>>
>>
>
>If this were to be restricted to kdump only, then it would be
>safe to add a field to the kdump_sub_header since it contains
>only one field.
>
>Whether it is necessary for diskdump is up to the diskdump
>maintainers to decide. But if they want to do it, they would
>have to come up with different manner of adding it to the
>disk_dump_sub_header structure, because of the way
>that it's declared, i.e., using "long elf_regs" as a placeholder
>to describe the start of the processor-dependent register
>set. They'd need to define a new disk_dump_sub_header
>version that alternatively would be used depending upon
>the header->version number.
We asked the diskdump developers (Indoh-san, Akiyama-san) how
to display diskdump's dump_level, and they said that changing
of file format is not necessary for displaying the dump_level.
And they gave us a good advice that the crash utility can get
diskdump's dump_level by using a symbol "dump_level" like the
following:
int get_dump_level(void)
{
if (symbol_exists("dump_level")) {
int dump_level;
readmem(symbol_value("dump_level"), KVADDR, &dump_level,
sizeof(dump_level), "dump_level",
FAULT_ON_ERROR);
return dump_level;
}
else
...
I think it is a good way because the crash utility can get diskdump's
dump_level of the existing dumpfile. I propose the following way for it.
Please let me know your opinion.
- If a symbol "dump_level" is present (diskdump), the dump_level is
taken from a symbol "dump_level".
- Else if a dumpfile is kdump's and header_version is 1 or more, the
dump_level taken from a new member "kdump_sub_header->dump_level".
Thanks
Ken'ichi Ohmichi
Both proposals sound good to me.
Just so that I can have it ready-to-go for you in the next crash release,
do you want to make the additional "dump_level" member an int or
a long?
Dave
3:02 a.m.
New subject: handling missing kdump pages in diskdump format
Hi Dave,
2007/04/05 09:04:53 -0400, Dave Anderson <anderson(a)redhat.com> wrote:
> >If this were to be restricted to kdump only, then it would
be
> >safe to add a field to the kdump_sub_header since it contains
> >only one field.
> >
> >Whether it is necessary for diskdump is up to the diskdump
> >maintainers to decide. But if they want to do it, they would
> >have to come up with different manner of adding it to the
> >disk_dump_sub_header structure, because of the way
> >that it's declared, i.e., using "long elf_regs" as a placeholder
> >to describe the start of the processor-dependent register
> >set. They'd need to define a new disk_dump_sub_header
> >version that alternatively would be used depending upon
> >the header->version number.
>
> We asked the diskdump developers (Indoh-san, Akiyama-san) how
> to display diskdump's dump_level, and they said that changing
> of file format is not necessary for displaying the dump_level.
> And they gave us a good advice that the crash utility can get
> diskdump's dump_level by using a symbol "dump_level" like the
> following:
>
> int get_dump_level(void)
> {
> if (symbol_exists("dump_level")) {
> int dump_level;
> readmem(symbol_value("dump_level"), KVADDR,
&dump_level,
> sizeof(dump_level), "dump_level",
> FAULT_ON_ERROR);
> return dump_level;
>
> }
> else
> ...
>
> I think it is a good way because the crash utility can get diskdump's
> dump_level of the existing dumpfile. I propose the following way for it.
> Please let me know your opinion.
>
> - If a symbol "dump_level" is present (diskdump), the dump_level is
> taken from a symbol "dump_level".
> - Else if a dumpfile is kdump's and header_version is 1 or more, the
> dump_level taken from a new member "kdump_sub_header->dump_level".
Both proposals sound good to me.
Just so that I can have it ready-to-go for you in the next crash release,
do you want to make the additional "dump_level" member an int or
a long?
I would like to add a member "dump_level" as int, because diskdump's
dump_level is defined as int. If it is defined as long, the crash
utility gets the dump_level with something else on 64bit system.
On x86_64 RHEL4 (2.6.9-42.EL), I tried getting the dump_level (15):
crash> rd dump_level
ffffffffa0140ba4: fffffffb0000000f ........
crash> rd -32 dump_level
ffffffffa0140ba4: 0000000f ....
crash>
And I think Bob's proposal (the excluded pages should not be accessed)
is worthy, too. I will add the following changes into makedumpfile.
makedumpfile:
- disk_dump_header->header_version is changed into 1 from 0.
- kdump_sub_header->dump_level is added as int, and makedumpfile sets
the value into it.
- The page descriptors (struct page_desc) of all the excluded zero-pages
point same zero-page. The crash utility can read zero-pages, even if
it cannot accesss the excluded pages.
Could you change the crash utility as the following ?
The crash utility:
- If disk_dump_header->signature is "KDUMP ", it checks its
header_version.
If it is 1 or more, the dump_level is taken from kdump_sub_header->dump_level,
and the excluded pages cannot be accessed (by default, or option ?).
Thanks
Ken'ichi Ohmichi
3:41 p.m.
New subject: handling missing kdump pages in diskdump format
On Thu, 2007-03-29 at 08:13 -0500, Dave Anderson wrote:
Ken'ichi Ohmichi wrote:
> I checked whether this change is correct by the following:
> (The following patches are attached with this mail)
> - makedumpfile-1.1.2 with "point_same_zero_page2.patch" creates a
dumpfile.
> - crash-4.0-3.21 with "not-access-excluded-page.patch" analyzes the
dumpfile.
> - The analysis result of the dumpfile is compared with /proc/vmcore's.
>
> And on i386 linux-2.6.19, I found the difference between the result
> of the dumpfile (excluding free pages) and /proc/vmcore's by subcommand
> "foreach bt".
> But by using crash-4.0-3.21 without "not-access-excluded-page.patch",
> there is not any difference. In a word, this difference happens due to
> considering the excluded pages as unaccess pages.
Just to clarify for those who probably aren't as confused as I was at
first:
This isn't a test of the zero page trick, because with the changes to
makedumpfile, zero pages are no longer actually excluded. (I read
"excluding free pages" but immediately thought "excluding zero pages"
and spent more than a few minutes checking how that could possibly have
happened.)
So this is apparently a case where a page excluded because it was
supposedly free is then maybe accessed by the back tracer while it might
be trying to read kernel text, right? But kernel text should never look
free, so I'm still puzzled. Did makedumpfile mis-identify a real page
as free, or is crash asking for pages it shouldn't be looking at during
backtrace?
Anyway, with my test dump on my x86_64 box, I don't get a case where
dumpfiles with excluded free pages produce different "foreach bt" output
than I get from the vmcore file. I tried -d16 and -d31 options. I do
get the expected excluded page message when I x/xg an excluded address,
just no problems with bt. So I can't help Dave with an example.
Bob Montgomery
4:09 p.m.
New subject: handling missing kdump pages in diskdump format
Bob Montgomery wrote:
On Thu, 2007-03-29 at 08:13 -0500, Dave Anderson wrote:
> Ken'ichi Ohmichi wrote:
> > I checked whether this change is correct by the following:
> > (The following patches are attached with this mail)
> > - makedumpfile-1.1.2 with "point_same_zero_page2.patch" creates a
dumpfile.
> > - crash-4.0-3.21 with "not-access-excluded-page.patch" analyzes the
dumpfile.
> > - The analysis result of the dumpfile is compared with /proc/vmcore's.
> >
> > And on i386 linux-2.6.19, I found the difference between the result
> > of the dumpfile (excluding free pages) and /proc/vmcore's by subcommand
> > "foreach bt".
> > But by using crash-4.0-3.21 without
"not-access-excluded-page.patch",
> > there is not any difference. In a word, this difference happens due to
> > considering the excluded pages as unaccess pages.
Just to clarify for those who probably aren't as confused as I was at
first:
This isn't a test of the zero page trick, because with the changes to
makedumpfile, zero pages are no longer actually excluded. (I read
"excluding free pages" but immediately thought "excluding zero
pages"
and spent more than a few minutes checking how that could possibly have
happened.)
So this is apparently a case where a page excluded because it was
supposedly free is then maybe accessed by the back tracer while it might
be trying to read kernel text, right? But kernel text should never look
free, so I'm still puzzled. Did makedumpfile mis-identify a real page
as free, or is crash asking for pages it shouldn't be looking at during
backtrace?
No -- it's kernel text that was marked as __init, so the page containing
it got freed and reallocated as a page that was purposely excluded.
The page originally contained the "start_kernel" __init function, which
only gets executed once by the first swapper thread.
The problem is that crash shouldn't be looking at that text location
when doing a backtrace on that PID 0, because it should have
stopped the trace as soon as it saw the "cpu_idle" stack reference.
I don't know why it's doing that -- I tried simulating Ken'ichi's vmcore
by forcibly returning an error if readmem() got a request for the
page originally containing "start_kernel", but the backtrace worked
OK -- even though I could see the "start_kernel" reference on
the stack when using "bt -t".
Anyway, that's why I've asked Ken'ichi if he can make his
vmlinux/vmcore pair available for me to debug.
Thanks,
Dave
Anyway, with my test dump on my x86_64 box, I don't get a case where
dumpfiles with excluded free pages produce different "foreach bt" output
than I get from the vmcore file. I tried -d16 and -d31 options. I do
get the expected excluded page message when I x/xg an excluded address,
just no problems with bt. So I can't help Dave with an example.
Bob Montgomery
10:59 a.m.
New subject: handling missing kdump pages in diskdump format
Dave Anderson wrote:
Bob Montgomery wrote:
> On Thu, 2007-03-29 at 08:13 -0500, Dave Anderson wrote:
> > Ken'ichi Ohmichi wrote:
>
> > > I checked whether this change is correct by the following:
> > > (The following patches are attached with this mail)
> > > - makedumpfile-1.1.2 with "point_same_zero_page2.patch" creates
a dumpfile.
> > > - crash-4.0-3.21 with "not-access-excluded-page.patch" analyzes
the dumpfile.
> > > - The analysis result of the dumpfile is compared with
/proc/vmcore's.
> > >
> > > And on i386 linux-2.6.19, I found the difference between the result
> > > of the dumpfile (excluding free pages) and /proc/vmcore's by
subcommand
> > > "foreach bt".
> > > But by using crash-4.0-3.21 without
"not-access-excluded-page.patch",
> > > there is not any difference. In a word, this difference happens due to
> > > considering the excluded pages as unaccess pages.
>
> Just to clarify for those who probably aren't as confused as I was at
> first:
>
> This isn't a test of the zero page trick, because with the changes to
> makedumpfile, zero pages are no longer actually excluded. (I read
> "excluding free pages" but immediately thought "excluding zero
pages"
> and spent more than a few minutes checking how that could possibly have
> happened.)
>
> So this is apparently a case where a page excluded because it was
> supposedly free is then maybe accessed by the back tracer while it might
> be trying to read kernel text, right? But kernel text should never look
> free, so I'm still puzzled. Did makedumpfile mis-identify a real page
> as free, or is crash asking for pages it shouldn't be looking at during
> backtrace?
>
No -- it's kernel text that was marked as __init, so the page containing
it got freed and reallocated as a page that was purposely excluded.
The page originally contained the "start_kernel" __init function, which
only gets executed once by the first swapper thread.
The problem is that crash shouldn't be looking at that text location
when doing a backtrace on that PID 0, because it should have
stopped the trace as soon as it saw the "cpu_idle" stack reference.
I don't know why it's doing that -- I tried simulating Ken'ichi's vmcore
by forcibly returning an error if readmem() got a request for the
page originally containing "start_kernel", but the backtrace worked
OK -- even though I could see the "start_kernel" reference on
the stack when using "bt -t".
Anyway, that's why I've asked Ken'ichi if he can make his
vmlinux/vmcore pair available for me to debug.
Thanks,
Dave
Actually, upon looking at a sample ELF-format dumpfile from Ken'ichi,
and hacking in a forced-readmem()-failure upon accessing the
page containing the start_kernel __init function, I can reproduce
the problem with the 2.6.19 kernel. But it is not a problem with
makedumpfile's excluded pages, but rather with the backtrace code's
framesize calculation of schedule() -- which is causing it skip over
the cpu_idle() ending point. It seems to be specific to 2.6.19 for
some reason. I'll look further into that issue.
I should note that the reading of the the start_kernel text is benign,
and only because the error message was added to read_diskdump() in
Ken'ichi's test crash utility is it even noticed. I've removed that error
message, and created a new one in readmem() specifically for attempts
to read excluded pages. However, if the readmem() caller has passed
in the QUIET flag to readmem() -- as is the case in the start_kernel
read -- the error message will not be shown unless the debug variable
is set non-zero. There are a number of place in the crash utility where
readmem()'s are not necessarily expected to work, and so it makes little
sense to always print error messages in read_diskdump().
One other thing I noticed with Ken'ichi's sample dumpfile. Again,
it's an ELF format file, but there are apparently some pages missing,
because some module pages (.eg, the page of the module that
contains the actual "module" structure) is zero'd out, and so you
get the "WARNING: cannot access vmalloc'd module memory"
during initialization.
For example, here's the modules list_head:
crash> p modules
modules = $2 = {
next = 0xf8b3f084,
prev = 0xf8828884
}
The first module in the list would be at 0xf8b3f080 (accounting
for the location of the module.list list_head struct). But the whole
page is filled with zeros:
crash> rd 0xf8b3f000 1024
f8b3f000: 00000000 00000000 00000000 00000000 ................
f8b3f010: 00000000 00000000 00000000 00000000 ................
f8b3f020: 00000000 00000000 00000000 00000000 ................
f8b3f030: 00000000 00000000 00000000 00000000 ................
f8b3f040: 00000000 00000000 00000000 00000000 ................
f8b3f050: 00000000 00000000 00000000 00000000 ................
...
If I look at the last module on the list at 0xf8828880, it
is contained in the dumpfile:
crash> module 0xf8828880
struct module {
state = MODULE_STATE_LIVE,
list = {
next = 0xc0382a90,
prev = 0xf882fc04
},
name =
"uhci_hcd\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
mkobj = {
kobj = {
k_name = 0xf88288cc "uhci_hcd",
name = "uhci_hcd\000\000\000\000\000\000\000\000\000\000\000",
...
and if I follow it back, several modules are there, but I bump into
another zero-filled module. (Thinking back, I can't say for sure whether
it's just bumping into the first one on the list or not...)
Anyway, it's cause for concern that the first one on the list is
either zero'd out, or not written to the dumpfile? With an ELF
dumpfile, I can't really tell.
Dave
11:15 p.m.
New subject: handling missing kdump pages in diskdump format
On Tue, Mar 20, 2007 at 11:18:02AM +0900, Ken'ichi Ohmichi wrote:
Hi Bob,
Thank you for the great report.
2007/03/14 08:40:16 +0530, Vivek Goyal <vgoyal(a)in.ibm.com> wrote:
>> =====================================
>> Can ELF Dumpfiles Solve This Problem?
>> =====================================
>>
>> To achieve correctness with ELF dumpfiles, one could perhaps remap the
>> four types of pages to the three types of ELF representations so that
>> "A) Not In The Address Space" and "B) Excluded Type" were
both mapped
>> to "1) Not In The Address Space". Then "C) Zero Content"
would map
>> to "2) Not In The File, Zero Fill". You would lose the ability to
>> know if a page were missing because it was never in the address space
>> in the first place, or because it was excluded because of its type.
>> But if you read a zero, you'd know it really was a zero.
>>
>
>I think this is the way to go. Why would I like to know if a page was
>never present or mkdumpfile filtered it out? I think we can live with that
>and lets just not create any sort of mapping for excluded pages in finally
>generated ELF headers.
In the above way, I worry that the crash utility cannot get the
relocatable information (machdep->machspec->phys_start) on ia64.
At ia64_calc_phys_start(), the crash utility gets phdr->p_paddr
of region 5 memory section as the relocatable information.
If the start of region 5 is excluded, the crash gets different
phdr->p_paddr in the above way.
Sorry, I don't understand it. vmcore ELF headers should give enough
information to determine what's the kernel's virtual and physical
address. Why should I have to have access to a particular page and because
of that take the wrong route and try to map filtered pages.
Is ia64 kernel mapped in region 5? If yes, I won't filter out kernel
pages?
If not, still I don't understand how does it help mapping non-kernel pages
and returning zeros for that?
Thanks
Vivek
6443
days inactive
6473
days old
devel@lists.crash-utility.osci.io
23 comments
4 participants
participants (4)
-
Bob Montgomery -
Dave Anderson -
Ken'ichi Ohmichi -
Vivek Goyal