Re: [Crash-utility] [patch]Crash can't process xen dump core files larger that 4GB.
by Dave Anderson
----- "Dave Anderson" <anderson(a)redhat.com> wrote:
> ----- "xiaowei hu" <xiaowei.hu(a)oracle.com> wrote:
>
> > Hi all,
> >
> > There is a bug when using crash to process the xen domU dump core that
> > larger that 4GB(it is found at processing a 10GB guest core dump file).
> > crash reporting this errors:
> > crash: cannot find mfn 8392757 (0x801035) in page index
> >
> >
> > crash: cannot read/find cr3 page
> >
> > this is caused by a var overflow,in the structure of
> > typedef struct xc_core_header {
> > unsigned int xch_magic;
> > unsigned int xch_nr_vcpus;
> > unsigned int xch_nr_pages;
> > unsigned int xch_ctxt_offset;
> > unsigned int xch_index_offset;
> > unsigned int xch_pages_offset;
> > } xc_core_header_t;
> >
> > the xch_ctxt_offset,xch_index_offset and xch_pages_offset mean the
> > offsets in the core dump file , when it is defined as unsingend
> > long ,that means the file can't be more that 4GB,so when processing with
> > core dump files that more than 4GB may have error (I encountered
> > overflow on that 10GB file),so changing those offset vars to unsigned
> > long ,make sure crash can seek to the right position.
> > btw,please reply directly to me ,I am not in the mail list.
> >
> >
> > Signed-off-by: Xiaowei Hu <xiaowei.hu(a)oracle.com>
> >
> >
> > diff -up crash-5.0.0/xendump.h.org crash-5.0.0/xendump.h
> > --- crash-5.0.0/xendump.h.org 2010-02-04 03:48:04.000000000 +0800
> > +++ crash-5.0.0/xendump.h 2010-02-04 05:41:27.000000000 +0800
> > @@ -28,9 +28,9 @@ typedef struct xc_core_header {
> > unsigned int xch_magic;
> > unsigned int xch_nr_vcpus;
> > unsigned int xch_nr_pages;
> > - unsigned int xch_ctxt_offset;
> > - unsigned int xch_index_offset;
> > - unsigned int xch_pages_offset;
> > + unsigned long xch_ctxt_offset;
> > + unsigned long xch_index_offset;
> > + unsigned long xch_pages_offset;
> > } xc_core_header_t;
> >
> > struct pfn_offset_cache {
>
> First question -- are you saying that the change above works for you?
>
> And second -- in your dumpfile, even with 10GB of memory, wouldn't
> the base offset value of all three indexes still fit well below
> the 4GB mark?
>
> The xc_core_header in crash is a copy of that found in "tools/libxc/xenctrl.h",
> and is presumptively the beginning/header of the dumpfile. And so making the
> wholesale change above breaks all earlier (?) versions.
>
> But what is confusing is that the latest/final version of "xenctrl.h" used in RHEL5
> (3.0.3 vintage), as well as the current version in Fedora (3.4.0-2.fc12) still use
> unsigned int offsets, and I just checked with one of our xen masters, and the Xensource
> git tree also still has unsigned int values in the header data
> structure:
>
> typedef struct xc_core_header {
> unsigned int xch_magic;
> unsigned int xch_nr_vcpus;
> unsigned int xch_nr_pages;
> unsigned int xch_ctxt_offset;
> unsigned int xch_index_offset;
> unsigned int xch_pages_offset;
> } xc_core_header_t;
>
> #define XC_CORE_MAGIC 0xF00FEBED
> #define XC_CORE_MAGIC_HVM 0xF00FEBEE
>
> Are your xen userspace tools an Oracle hybrid?
Ah -- it's becoming clearer now...
The evolution of the various xendump formats is the cause for confusion
and the issue at hand.
In the beginning, the "xm dump-core" facility used its own unique dumpfile
format, where the xc_core_header shown above was at the beginning
of the dumpfile and served as its primary header.
Much later, "xm dump-core" started using an ELF format, where it
carried forward 3 of the old xc_core_header fields above into either
this ELF note:
struct xen_dumpcore_elfnote_header_desc {
uint64_t xch_magic;
uint64_t xch_nr_vcpus;
uint64_t xch_nr_pages;
uint64_t xch_page_size;
};
or into one of several ELF section headers. The remaining 3 "offset" fields
are stored like so:
xch_ctxt_offset: in the ".xen_prstatus" ELF section header
xch_index_offset: in the ".xen_pfn" or ".xen_p2m" ELF section header
depending whether it's fully-virtualized or
paravirtualized.
xch_pages_offset: in the ".xen_pages" ELF section header
The offsets are in the ELF section headers are of "sh_offset" fields
of the Elf64_Shdr (or Elf32_Shdr if ELFCLASS32):
typedef struct
{
Elf64_Word sh_name; /* Section name (string tbl index) */
Elf64_Word sh_type; /* Section type */
Elf64_Xword sh_flags; /* Section flags */
Elf64_Addr sh_addr; /* Section virtual addr at execution */
Elf64_Off sh_offset; /* Section file offset */
Elf64_Xword sh_size; /* Section size in bytes */
Elf64_Word sh_link; /* Link to another section */
Elf64_Word sh_info; /* Additional section information */
Elf64_Xword sh_addralign; /* Section alignment */
Elf64_Xword sh_entsize; /* Entry size if section holds table */
} Elf64_Shdr;
FWIW, I don't know (or recall) whether ELFCLASS32 is ever used, even with 32-bit
xen hosts/guests, because the "sh_offset" in the Elf32_Shdr is of type
Elf32_Off, which is 32-bits:
/* Type of file offsets. */
typedef uint32_t Elf32_Off;
typedef uint64_t Elf64_Off;
Anyway, the problem is that the crash utility started using the old xc_core_header
data structure when it was the only header. When they started using ELF format
dumpfiles, the sh_offset values from the ELF section headers were copied into
the old xc_core_header data structure in the crash utility so that the old code
base could still be used. But if any of the sh_offset values overflowed into
the upper 32-bits, then they would be truncated when the copy was made.
In any case, getting back to the crash utility issue, the patch that you
proposed cannot be used alone because it will break backwards-compatibility.
What could be done is to have the xc_core_verify() initialization code read
the dumpfile header into an "original" xc_core_header structure type, verify it
as one of the "old-style" dumpfiles, but then store the offsets into your
updated xc_core_header structure.
Dave
The xc_core_header above
16 years
答复: Re: [patch]Crash can't process xen dump core files larger that 4GB.
by Xiaowei Hu
----- "xiaowei hu" <xiaowei.hu(a)oracle.com> wrote:
> Hi all,
>
> There is a bug when using crash to process the xen domU dump core that
> larger that 4GB(it is found at processing a 10GB guest core dump file).
> crash reporting this errors:
> crash: cannot find mfn 8392757 (0x801035) in page index
>
>
> crash: cannot read/find cr3 page
>
> this is caused by a var overflow,in the structure of
> typedef struct xc_core_header {
> unsigned int xch_magic;
> unsigned int xch_nr_vcpus;
> unsigned int xch_nr_pages;
> unsigned int xch_ctxt_offset;
> unsigned int xch_index_offset;
> unsigned int xch_pages_offset;
> } xc_core_header_t;
>
> the xch_ctxt_offset,xch_index_offset and xch_pages_offset mean the
> offsets in the core dump file , when it is defined as unsingend
> long ,that means the file can't be more that 4GB,so when processing with
> core dump files that more than 4GB may have error (I encountered
> overflow on that 10GB file),so changing those offset vars to unsigned
> long ,make sure crash can seek to the right position.
> btw,please reply directly to me ,I am not in the mail list.
>
>
> Signed-off-by: Xiaowei Hu <xiaowei.hu(a)oracle.com>
>
>
> diff -up crash-5.0.0/xendump.h.org crash-5.0.0/xendump.h
> --- crash-5.0.0/xendump.h.org 2010-02-04 03:48:04.000000000 +0800
> +++ crash-5.0.0/xendump.h 2010-02-04 05:41:27.000000000 +0800
> @@ -28,9 +28,9 @@ typedef struct xc_core_header {
> unsigned int xch_magic;
> unsigned int xch_nr_vcpus;
> unsigned int xch_nr_pages;
> - unsigned int xch_ctxt_offset;
> - unsigned int xch_index_offset;
> - unsigned int xch_pages_offset;
> + unsigned long xch_ctxt_offset;
> + unsigned long xch_index_offset;
> + unsigned long xch_pages_offset;
> } xc_core_header_t;
>
> struct pfn_offset_cache {
>First question -- are you saying that the change above works for you?
yes, this change works for me on a 10GB dump core file,whose .xen_p2m segment's offset at
0x280005000, this offset can't be stored in a unsinged int var.
>And second -- in your dumpfile, even with 10GB of memory, wouldn't
>the base offset value of all three indexes still fit well below
>the 4GB mark?
actually from the xen-dump-core document the .xen_p2m segment should be located before
the .xen_pages segment, in this order ,there is should not have problem.
but according the segment table read by readelf,I found the core dump file have the xen_p2m
segment located at offset 0x2800025000 after the .xen_pages segment,beyond the 4GB mark.
>The xc_core_header in crash is a copy of that found in "tools/libxc/xenctrl.h",
>and is presumptively the beginning/header of the dumpfile. And so making the
>wholesale change above breaks all earlier (?) versions.
>But what is confusing is that the latest/final version of "xenctrl.h" used in RHEL5
>(3.0.3 vintage), as well as the current version in Fedora (3.4.0-2.fc12) still use
>unsigned int offsets, and I just checked with one of our xen masters, and the Xensource
>git tree also still has unsigned int values in the header data structure:
>typedef struct xc_core_header {
> unsigned int xch_magic;
> unsigned int xch_nr_vcpus;
> unsigned int xch_nr_pages;
> unsigned int xch_ctxt_offset;
> unsigned int xch_index_offset;
> unsigned int xch_pages_offset;
>} xc_core_header_t;
>#define XC_CORE_MAGIC 0xF00FEBED
>#define XC_CORE_MAGIC_HVM 0xF00FEBEE
>Are your xen userspace tools an Oracle hybrid?
yes, the core dump file is generated on oracle virtualization server.But I did not check the ovm
source code for changes of this header data structure.will check it and replay again tommorrow.
>Dave
thanks
xiaowei
16 years
Re: [Crash-utility] [patch]Crash can't process xen dump core files larger that 4GB.
by Dave Anderson
----- "xiaowei hu" <xiaowei.hu(a)oracle.com> wrote:
> Hi all,
>
> There is a bug when using crash to process the xen domU dump core that
> larger that 4GB(it is found at processing a 10GB guest core dump file).
> crash reporting this errors:
> crash: cannot find mfn 8392757 (0x801035) in page index
>
>
> crash: cannot read/find cr3 page
>
> this is caused by a var overflow,in the structure of
> typedef struct xc_core_header {
> unsigned int xch_magic;
> unsigned int xch_nr_vcpus;
> unsigned int xch_nr_pages;
> unsigned int xch_ctxt_offset;
> unsigned int xch_index_offset;
> unsigned int xch_pages_offset;
> } xc_core_header_t;
>
> the xch_ctxt_offset,xch_index_offset and xch_pages_offset mean the
> offsets in the core dump file , when it is defined as unsingend
> long ,that means the file can't be more that 4GB,so when processing with
> core dump files that more than 4GB may have error (I encountered
> overflow on that 10GB file),so changing those offset vars to unsigned
> long ,make sure crash can seek to the right position.
> btw,please reply directly to me ,I am not in the mail list.
>
>
> Signed-off-by: Xiaowei Hu <xiaowei.hu(a)oracle.com>
>
>
> diff -up crash-5.0.0/xendump.h.org crash-5.0.0/xendump.h
> --- crash-5.0.0/xendump.h.org 2010-02-04 03:48:04.000000000 +0800
> +++ crash-5.0.0/xendump.h 2010-02-04 05:41:27.000000000 +0800
> @@ -28,9 +28,9 @@ typedef struct xc_core_header {
> unsigned int xch_magic;
> unsigned int xch_nr_vcpus;
> unsigned int xch_nr_pages;
> - unsigned int xch_ctxt_offset;
> - unsigned int xch_index_offset;
> - unsigned int xch_pages_offset;
> + unsigned long xch_ctxt_offset;
> + unsigned long xch_index_offset;
> + unsigned long xch_pages_offset;
> } xc_core_header_t;
>
> struct pfn_offset_cache {
First question -- are you saying that the change above works for you?
And second -- in your dumpfile, even with 10GB of memory, wouldn't
the base offset value of all three indexes still fit well below
the 4GB mark?
The xc_core_header in crash is a copy of that found in "tools/libxc/xenctrl.h",
and is presumptively the beginning/header of the dumpfile. And so making the
wholesale change above breaks all earlier (?) versions.
But what is confusing is that the latest/final version of "xenctrl.h" used in RHEL5
(3.0.3 vintage), as well as the current version in Fedora (3.4.0-2.fc12) still use
unsigned int offsets, and I just checked with one of our xen masters, and the Xensource
git tree also still has unsigned int values in the header data structure:
typedef struct xc_core_header {
unsigned int xch_magic;
unsigned int xch_nr_vcpus;
unsigned int xch_nr_pages;
unsigned int xch_ctxt_offset;
unsigned int xch_index_offset;
unsigned int xch_pages_offset;
} xc_core_header_t;
#define XC_CORE_MAGIC 0xF00FEBED
#define XC_CORE_MAGIC_HVM 0xF00FEBEE
Are your xen userspace tools an Oracle hybrid?
Dave
16 years
[patch]Crash can't process xen dump core files larger that 4GB.
by xiaowei hu
Hi all,
There is a bug when using crash to process the xen domU dump core that
larger that 4GB(it is found at processing a 10GB guest core dump file).
crash reporting this errors:
crash: cannot find mfn 8392757 (0x801035) in page index
crash: cannot read/find cr3 page
this is caused by a var overflow,in the structure of
typedef struct xc_core_header {
unsigned int xch_magic;
unsigned int xch_nr_vcpus;
unsigned int xch_nr_pages;
unsigned int xch_ctxt_offset;
unsigned int xch_index_offset;
unsigned int xch_pages_offset;
} xc_core_header_t;
the xch_ctxt_offset,xch_index_offset and xch_pages_offset mean the
offsets in the core dump file , when it is defined as unsingend
long ,that means the file can't be more that 4GB,so when processing with
core dump files that more than 4GB may have error (I encountered
overflow on that 10GB file),so changing those offset vars to unsigned
long ,make sure crash can seek to the right position.
btw,please reply directly to me ,I am not in the mail list.
Signed-off-by: Xiaowei Hu <xiaowei.hu(a)oracle.com>
diff -up crash-5.0.0/xendump.h.org crash-5.0.0/xendump.h
--- crash-5.0.0/xendump.h.org 2010-02-04 03:48:04.000000000 +0800
+++ crash-5.0.0/xendump.h 2010-02-04 05:41:27.000000000 +0800
@@ -28,9 +28,9 @@ typedef struct xc_core_header {
unsigned int xch_magic;
unsigned int xch_nr_vcpus;
unsigned int xch_nr_pages;
- unsigned int xch_ctxt_offset;
- unsigned int xch_index_offset;
- unsigned int xch_pages_offset;
+ unsigned long xch_ctxt_offset;
+ unsigned long xch_index_offset;
+ unsigned long xch_pages_offset;
} xc_core_header_t;
struct pfn_offset_cache {
16 years
Re: [Crash-utility] Crash Subscription - Something Wrong
by Piet Delaney
Dave Anderson wrote:
> Piet Delaney wrote:
>> Hi Dave [I moved]:
>>
>> I subscribed to your crash mailing list last evening and didn't receive
>> anything yet.
>> Looks like there was a posting today.
>
> I remember approving your subscription request. Looking at the current
> list of subscribers, I see these two:
>
> Piet.Delaney(a)tensilica.com
> piet(a)bluelane.com
>
>> How much work do you think it would take to add the xtensa architecture
>> to crash?
>> From our IA64 work together I recall it mostly being the backtrace code
>> needing work.
>
> That's certainly a big part of it, but that's only the "gravy" that needs
> to be done after all of the foundational work is done. New architectures
> basically require:
>
> 1. cloning the arch-specific parts of configure.c so that there
> can be "#ifdef XTENSA" parts of the code.
> 2. creating a new "xtensa.c" file that satisfies all of the
> requirements for the generic machdep-><architecture-data> and
> machdep-><architecture-function> fields used throughout the
> common code. The crash sources do contain a template PLATFORM.c
> file, which is a bit dated and will most likely not have everything
> needed, but it's a good place to start. You would basically copy
> PLATFORM.c to xtensa.c, and then do a "s/PLATFORM/xtensa/" on the
> file, and then start working on the specifics.
I'll start work on xtensa.c next now that configure.c seems to be fine.
> 3. in conjunction with step 2, satisfying all of the "common" #define's
> in defs.h.
> 4. After 1, 2 and 3 are done, update the Makefile to compile the
> new file, and as it continually fails due to missing stuff, fix then
> one step at a time.
> 5. At that point, you can start working on stuff like the backtrace
> implementation.
>
> Although -- now that I think of it -- none of the above will be worth
> a damn without gdb support of the architecture. And gdb-6.1 certainly
> doesn't have it unless it's a clone of some pre-existing architecture.
With your new crash 5.0 and it's use of gdb-8.0 that's no longer a problem.
>
> So that's a whole other can of worms. Updating the version of gdb to
> "merge" with the crash sources is yet another massive, always prone-to-error,
> always finding-something-that-no-long-applies-anymore, situation.
Besides my maintaining a subscription to your mailing list I've created a crash-xtensa
mailing list at linux-xtensa.org and an associated crash-xtensa-commits for logging the
git repository changes. Your welcome to be a list administrators if your interested to to
recommend when to push changes/patches to your repository.
Should be interesting but likely a rather slow effort; lots of other stuff on my plate.
-piet
>
> Dave
>
>> Some the the 'vm' command functionality would be helpful right now to debug
>> a cache/tldb bug that I'm looking at.
>>
>> -piet
>
16 years
Re: [Crash-utility] segv in crash-5.0.0
by Dave Anderson
----- "Dave Anderson" <anderson(a)redhat.com> wrote:
> ----- "Bob Montgomery" <bob.montgomery(a)hp.com> wrote:
>
> > I accidentally tried to dump a struct from a bogus pointer while using
> > crash-5.0.0 on x86-64.
... [ snip ] ...
> > Enough to go on? Already known?
>
> Not already known...
>
> But I can reproduce it (at least with some bogus addresses) -- I'll take
> a look at it tomorrow...
Caused by a slight change in the crash/gdb-7.0 exception handling.
Simple fix attached...
Thanks,
Dave
16 years
Re: [Crash-utility] segv in crash-5.0.0
by Dave Anderson
----- "Bob Montgomery" <bob.montgomery(a)hp.com> wrote:
> I accidentally tried to dump a struct from a bogus pointer while using
> crash-5.0.0 on x86-64.
>
> In crash-4.1.1, the result was:
> crash> struct bnx2 0xffffc90006b000cf
> struct bnx2 struct: invalid kernel virtual address: ffffc90006b000cf
> type: "gdb_readmem_callback"
> Cannot access memory at address 0xffffc90006b000cf
> crash>
>
> On crash-5.0.0, the result was:
> crash-5.0> struct bnx2 0xffffc90006b000cf
> struct bnx2 struct: invalid kernel virtual address: ffffc90006b000cf
> type: "gdb_readmem_callback"
> *** glibc detected *** crash-5.0: double free or corruption (!prev):
> 0x0000000006f94e60 ***
> gdb called without error_hook: Cannot access memory at address
> 0xffffc90006b000cf
> <segmentation violation in gdb>
>
> [[ Here the process hung, and I had to kill -9 it ]]
>
>
> While running crash-5.0.0 under gdb, I tried some non-struct accesses of
> the location first:
> crash> rd 0xffffc90006b000cf 10
> rd: invalid kernel virtual address: ffffc90006b000cf type: "64-bit KVADDR"
> crash> x/xg 0xffffc90006b000cf
> 0xffffc90006b000cf: gdb: invalid kernel virtual address:
> ffffc90006b000cf type: "gdb_readmem_callback"
> Cannot access memory at address 0xffffc90006b000cf
> gdb: gdb request failed: x/xg
> crash>
>
> But with the struct access:
>
> crash> struct bnx2 0xffffc90006b000cf
> struct bnx2 struct: invalid kernel virtual address: ffffc90006b000cf type: "gdb_readmem_callback"
> gdb called without error_hook: Cannot access memory at address 0xffffc90006b000cf
> *** glibc detected *** /home/bobm/Crash/crash-5.0.0/crash: double free
> or corruption (!prev): 0x0000000007144210 ***
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007fd3c65781af in ?? () from /lib/libgcc_s.so.1
> (gdb)
> (gdb) bt
> #0 0x00007fd3c65781af in ?? () from /lib/libgcc_s.so.1
> #1 0x00007fd3c6578a7b in _Unwind_Backtrace () from /lib/libgcc_s.so.1
> #2 0x00007fd3cda735ae in backtrace () from /lib/libc.so.6
> #3 0x00007fd3cda013bc in __libc_message () from /lib/libc.so.6
> #4 0x00007fd3cda06948 in malloc_printerr () from /lib/libc.so.6
> #5 0x00007fd3cda08a56 in free () from /lib/libc.so.6
> #6 0x000000000058807a in parse_exp_in_context (stringptr=0x7fffb04cdc20,
> block=<value optimized out>, comma=<value optimized out>,
> void_context_p=32723, out_subexp=0x7fffb04cdba0) at parse.c:1101
> #7 0x0000001d06b000cf in ?? ()
> #8 0x0000000000000000 in ?? ()
>
> Enough to go on? Already known?
Not already known...
But I can reproduce it (at least with some bogus addresses) -- I'll take
a look at it tomorrow...
Thanks,
Dave
16 years
segv in crash-5.0.0
by Bob Montgomery
I accidentally tried to dump a struct from a bogus pointer while using
crash-5.0.0 on x86-64.
In crash-4.1.1, the result was:
crash> struct bnx2 0xffffc90006b000cf
struct bnx2 struct: invalid kernel virtual address: ffffc90006b000cf
type: "gdb_readmem_callback"
Cannot access memory at address 0xffffc90006b000cf
crash>
On crash-5.0.0, the result was:
crash-5.0> struct bnx2 0xffffc90006b000cf
struct bnx2 struct: invalid kernel virtual address: ffffc90006b000cf
type: "gdb_readmem_callback"
*** glibc detected *** crash-5.0: double free or corruption (!prev):
0x0000000006f94e60 ***
gdb called without error_hook: Cannot access memory at address
0xffffc90006b000cf
<segmentation violation in gdb>
[[ Here the process hung, and I had to kill -9 it ]]
While running crash-5.0.0 under gdb, I tried some non-struct accesses of
the location first:
crash> rd 0xffffc90006b000cf 10
rd: invalid kernel virtual address: ffffc90006b000cf type: "64-bit
KVADDR"
crash> x/xg 0xffffc90006b000cf
0xffffc90006b000cf: gdb: invalid kernel virtual address:
ffffc90006b000cf type: "gdb_readmem_callback"
Cannot access memory at address 0xffffc90006b000cf
gdb: gdb request failed: x/xg
crash>
But with the struct access:
crash> struct bnx2 0xffffc90006b000cf
struct bnx2 struct: invalid kernel virtual address: ffffc90006b000cf
type: "gdb_readmem_callback"
gdb called without error_hook: Cannot access memory at address
0xffffc90006b000cf
*** glibc detected *** /home/bobm/Crash/crash-5.0.0/crash: double free
or corruption (!prev): 0x0000000007144210 ***
Program received signal SIGSEGV, Segmentation fault.
0x00007fd3c65781af in ?? () from /lib/libgcc_s.so.1
(gdb)
(gdb) bt
#0 0x00007fd3c65781af in ?? () from /lib/libgcc_s.so.1
#1 0x00007fd3c6578a7b in _Unwind_Backtrace () from /lib/libgcc_s.so.1
#2 0x00007fd3cda735ae in backtrace () from /lib/libc.so.6
#3 0x00007fd3cda013bc in __libc_message () from /lib/libc.so.6
#4 0x00007fd3cda06948 in malloc_printerr () from /lib/libc.so.6
#5 0x00007fd3cda08a56 in free () from /lib/libc.so.6
#6 0x000000000058807a in parse_exp_in_context
(stringptr=0x7fffb04cdc20,
block=<value optimized out>, comma=<value optimized out>,
void_context_p=32723, out_subexp=0x7fffb04cdba0) at parse.c:1101
#7 0x0000001d06b000cf in ?? ()
#8 0x0000000000000000 in ?? ()
Enough to go on? Already known?
Thanks,
Bob Montgomery
16 years