3:19 p.m.
Hello Ken'ichi,
First let me say that I very much appreciate your looking into
this issue.
However, there is a problem with your patch such that it won't work
with the sample Fedora kernel that I was planning to use as a test case.
That kernel is this Fedora version:
Linux version 2.6.27-0.244.rc2.git1.fc10.x86_64
When I run with your patches, I still get the same initialization
time failure:
crash: cannot resolve: "end_pfn"
With the Fedora kernel, your changes to x86_64_init() are not executed because
the VM_2_6_11 flag does not get set here, but rather VM_XEN get set:
case PRE_GDB:
if (!(machdep->flags & VM_FLAGS)) {
if (symbol_exists("xen_start_info")) {
if (symbol_exists("low_pml4") &&
symbol_exists("swap_low_mappings"))
machdep->flags |= VM_XEN_RHEL4;
else
==============> machdep->flags |= VM_XEN;
} else if (symbol_exists("boot_vmalloc_pgt"))
machdep->flags |= VM_ORIG;
else
machdep->flags |= VM_2_6_11;
}
And that's because the Fedora kernel contains a "xen_start_info" symbol,
and so it presumes it's a xen kernel. Even if I force it to use
VM_2_6_11 with "--machdep vm=2.6.11" on the command line, it still
fails the same way in kernel_init() because, again, it thinks it's a xen
kernel:
if (symbol_exists("xen_start_info")) {
kt->flags |= ARCH_XEN;
if (!(kt->xen_flags & (SHADOW_PAGE_TABLES|CANONICAL_PAGE_TABLES)))
kt->xen_flags |= WRITABLE_PAGE_TABLES;
if (symbol_exists("phys_to_machine_mapping"))
get_symbol_data("phys_to_machine_mapping",
sizeof(ulong),
&kt->phys_to_machine_mapping);
else if (!(kt->xen_flags & CANONICAL_PAGE_TABLES)) {
kt->xen_flags &= ~WRITABLE_PAGE_TABLES;
kt->xen_flags |= SHADOW_PAGE_TABLES;
}
if (machine_type("X86"))
get_symbol_data("max_pfn", sizeof(ulong),
&kt->p2m_table_size);
if (machine_type("X86_64"))
===========> get_symbol_data("end_pfn", sizeof(ulong),
&kt->p2m_table_size);
if ((kt->m2p_page = (char *)malloc(PAGESIZE())) == NULL)
error(FATAL, "cannot malloc m2p page.");
}
I made a preliminary inquiry as to why a bare-metal kernel would have
a bunch of (unused) xen* symbols built into it, and as I understand it,
the pv_ops implementation allows for the same kernel (vmlinux) to be
used for bare-metal, or for xen/pv_ops, and for that matter kvm/pv_ops
and VMI/pv_ops (?).
Now, clearly the linux-2.6.27-rc5 vmlinux that you used for testing your
patch does *not* have the "xen_start_info" symbol contained in it, because
if it did, you would have run into the same problem as me. I'm guessing
that all of the xen symbols I seen in my kernel are due to a CONFIG-option
used by the Fedora kernel build? And perhaps the same thing is done for KVM?
(VMI?) By any chance, do you know exactly how the pv_ops implementations fit
together?
In any case, the benefit of the pv_ops implementation is supposed to be
the fact that the same vmlinux kernel can be used for bare-metal or virtualized
kernels, but it potentially makes things more difficult for the crash
utility. But given that the bare-metal/virualized kernels are identical, then
presumably the kernel's PAGE_OFFSET would be identical, and that's what your
patch
is addressing. However, crash will also need a way to determine whether
it's a xen kernel or not. For example, look at the crash sources for all
users of the XEN() macro.
So I'm thinking that there may have to be something like a new
virtualization_init() function called very early on that can determine
whether this kernel is a newer pv_ops kernel with xen, kvm, etc. possibly
built-in? So in your linux-2.6.27-rc5 vmlinux, the new function would see the
pv_ops symbols but no xen symbols, so it could set some "bare-metal" flag.
But in the Fedora kernel's case, it would see both pv_ops and xen symbols,
and then would have to actually read (if possible) the xen_start_info pointer
to see if it's been initialized to something -- hopefully without getting into
a chicken-and-egg situation. Worst case, it may require new crash command line
arguments such as --xen or --kvm. But I'm hoping to avoid that if at all possible.
Whatever gets done, it has the potential to be pretty ugly...
Do you have any thoughts on the matter?
Dave
----- "Ken'ichi Ohmichi" <oomichi(a)mxs.nes.nec.co.jp> wrote:
Hi,
Since linux-2.6.27 of x86_64, PAGE_OFFSET has been changed to
0xffff880000000000 from 0xffff810000000000.
This patch catches up this change.
I tested it on linux-2.6.27-rc5, and it works fine.
Thanks
Ken'ichi Ohmichi
Signed-off-by: Ken'ichi Ohmichi <oomichi(a)mxs.nes.nec.co.jp>
---
diff -rpuN crash-4.0-7.1.orig/defs.h crash-4.0-7.1/defs.h
--- crash-4.0-7.1.orig/defs.h 2008-09-01 20:19:06.000000000 +0900
+++ crash-4.0-7.1/defs.h 2008-09-01 20:22:07.000000000 +0900
@@ -2124,6 +2124,8 @@ struct load_module {
#define VMEMMAP_VADDR_2_6_24 0xffffe20000000000
#define VMEMMAP_END_2_6_24 0xffffe2ffffffffff
+#define PAGE_OFFSET_2_6_27 0xffff880000000000
+
#define USERSPACE_TOP_XEN 0x0000800000000000
#define PAGE_OFFSET_XEN 0xffff880000000000
#define VMALLOC_START_ADDR_XEN 0xffffc20000000000
diff -rpuN crash-4.0-7.1.orig/x86_64.c crash-4.0-7.1/x86_64.c
--- crash-4.0-7.1.orig/x86_64.c 2008-09-01 20:19:06.000000000 +0900
+++ crash-4.0-7.1/x86_64.c 2008-09-01 20:32:14.000000000 +0900
@@ -178,7 +178,6 @@ x86_64_init(int when)
case VM_2_6_11:
/* 2.6.11 layout */
machdep->machspec->userspace_top = USERSPACE_TOP_2_6_11;
- machdep->machspec->page_offset = PAGE_OFFSET_2_6_11;
machdep->machspec->vmalloc_start_addr =
VMALLOC_START_ADDR_2_6_11;
machdep->machspec->vmalloc_end = VMALLOC_END_2_6_11;
machdep->machspec->modules_vaddr = MODULES_VADDR_2_6_11;
@@ -190,6 +189,13 @@ x86_64_init(int when)
if (symbol_exists("vmemmap_populate"))
machdep->flags |= VMEMMAP;
+ if (symbol_exists("end_pfn"))
+ /* 2.6.11 layout */
+ machdep->machspec->page_offset = PAGE_OFFSET_2_6_11;
+ else
+ /* 2.6.27 layout */
+ machdep->machspec->page_offset = PAGE_OFFSET_2_6_27;
+
machdep->uvtop = x86_64_uvtop_level4;
break;
---
12:28 a.m.
New subject: [PATCH 1/2] Add x86_64 linux-2.6.27 support: Catch up the PAGE_OFFSET change.
Hi Dave,
Thank you for checking my patch.
Dave Anderson wrote:
However, there is a problem with your patch such that it won't
work
with the sample Fedora kernel that I was planning to use as a test case.
That kernel is this Fedora version:
Linux version 2.6.27-0.244.rc2.git1.fc10.x86_64
When I run with your patches, I still get the same initialization
time failure:
crash: cannot resolve: "end_pfn"
With the Fedora kernel, your changes to x86_64_init() are not executed because
the VM_2_6_11 flag does not get set here, but rather VM_XEN get set:
[snip]
Now, clearly the linux-2.6.27-rc5 vmlinux that you used for testing
your
patch does *not* have the "xen_start_info" symbol contained in it, because
if it did, you would have run into the same problem as me. I'm guessing
that all of the xen symbols I seen in my kernel are due to a CONFIG-option
used by the Fedora kernel build? And perhaps the same thing is done for KVM?
(VMI?) By any chance, do you know exactly how the pv_ops implementations fit
together?
I could reproduce the same problem.
As you guessed, my vmlinux had been built without the following CONFIG items:
- CONFIG_PARAVIRT_GUEST=y
- CONFIG_XEN=y
- CONFIG_XEN_MAX_DOMAIN_MEMORY=32
- CONFIG_XEN_SAVE_RESTORE=y
- CONFIG_KVM_CLOCK=y
- CONFIG_KVM_GUEST=y
- CONFIG_PARAVIRT=y
- CONFIG_PARAVIRT_CLOCK=y
Then, I built a vmlinux with these items, and I saw the same problem that
the crash with my patch cannot run when the initialization time:
crash: cannot resolve: "end_pfn"
We should fix it.
In any case, the benefit of the pv_ops implementation is supposed to
be
the fact that the same vmlinux kernel can be used for bare-metal or virtualized
kernels, but it potentially makes things more difficult for the crash
utility. But given that the bare-metal/virualized kernels are identical, then
presumably the kernel's PAGE_OFFSET would be identical, and that's what your
patch
is addressing. However, crash will also need a way to determine whether
it's a xen kernel or not. For example, look at the crash sources for all
users of the XEN() macro.
So I'm thinking that there may have to be something like a new
virtualization_init() function called very early on that can determine
whether this kernel is a newer pv_ops kernel with xen, kvm, etc. possibly
built-in? So in your linux-2.6.27-rc5 vmlinux, the new function would see the
pv_ops symbols but no xen symbols, so it could set some "bare-metal" flag.
But in the Fedora kernel's case, it would see both pv_ops and xen symbols,
and then would have to actually read (if possible) the xen_start_info pointer
to see if it's been initialized to something -- hopefully without getting into
a chicken-and-egg situation. Worst case, it may require new crash command line
arguments such as --xen or --kvm. But I'm hoping to avoid that if at all possible.
I think that a new virtualization_init() function is a good idea.
How about checking whether "Xen" exists in PT_NOTE section of an vmcore
at virtualization_init()? I feel that it is safer than reading a xen_start_info
pointer.
$ readelf -n vmcore
Notes at offset 0x00000120 with length 0x00001388:
Owner Data size Description
CORE 0x00000150 NT_PRSTATUS (prstatus structure)
Xen 0x00000020 Unknown note type: (0x01000002) <----- HERE
CORE 0x00000150 NT_PRSTATUS (prstatus structure)
Xen 0x00000020 Unknown note type: (0x01000002) <----- HERE
Xen 0x00000050 Unknown note type: (0x01000001) <----- HERE
VMCOREINFO_XEN 0x00000fe4 Unknown note type: (0x00000000)
$
Thanks
Ken'ichi Ohmichi
6:29 a.m.
New subject: [PATCH 1/2] Add x86_64 linux-2.6.27 support: Catch up the PAGE_OFFSET change.
Ken'ichi Ohmichi wrote:
Hi Dave,
[ snip ]
I think that a new virtualization_init() function is a good idea.
How about checking whether "Xen" exists in PT_NOTE section of an vmcore
at virtualization_init()? I feel that it is safer than reading a xen_start_info
pointer.
$ readelf -n vmcore
Notes at offset 0x00000120 with length 0x00001388:
Owner Data size Description
CORE 0x00000150 NT_PRSTATUS (prstatus structure)
Xen 0x00000020 Unknown note type: (0x01000002) <----- HERE
CORE 0x00000150 NT_PRSTATUS (prstatus structure)
Xen 0x00000020 Unknown note type: (0x01000002) <----- HERE
Xen 0x00000050 Unknown note type: (0x01000001) <----- HERE
VMCOREINFO_XEN 0x00000fe4 Unknown note type: (0x00000000)
$
Well there may be a couple issues with that approach. First, we need
a way to handle/recognize it on live systems. And second, I believe we'll need
a way to differentiate a pv_ops/xen kernel from a "dedicated" xen kernel.
And my guess is that the vmcore notes will not differentiate between the
two "flavors" of xen kernel (unless there's an entry in the VMCOREINFO_XEN
that will somehow differentiate the two).
I'm thinking now that a pv_ops/xen kernel should not be simply handled by
making the current XEN() macro usable. Looking at the various usages of XEN(),
I think that many of them may *only* be applicable to dedicated xen kernels.
So I believe there should be a new PV_XEN() macro, and ultimately PV_KVM()
and PV_VMI() macros. Given that all of the pv_ops kernels are the "same"
vmlinux, it's possible that in some ways things may be simplified. But I
strongly doubt that the use of the current XEN() macro will work.
So I think it's best to take it one step at a time. The first step will be
to get crash working with 2.6.27 bare-metal kernels, and disallow the setting
of the ARCH_XEN flag used by the XEN() macro. Until I actually see a "real"
pv_ops/xen kernel dumpfile, it's not even worth pre-coding for it.
Dave
3:52 a.m.
New subject: [PATCH 1/2] Add x86_64 linux-2.6.27 support: Catch up the PAGE_OFFSET change.
Hello Dave,
Dave Anderson wrote:
> I think that a new virtualization_init() function is a good
idea.
>
> How about checking whether "Xen" exists in PT_NOTE section of an vmcore
> at virtualization_init()? I feel that it is safer than reading a xen_start_info
> pointer.
>
> $ readelf -n vmcore
> Notes at offset 0x00000120 with length 0x00001388:
> Owner Data size Description
> CORE 0x00000150 NT_PRSTATUS (prstatus structure)
> Xen 0x00000020 Unknown note type: (0x01000002) <----- HERE
> CORE 0x00000150 NT_PRSTATUS (prstatus structure)
> Xen 0x00000020 Unknown note type: (0x01000002) <----- HERE
> Xen 0x00000050 Unknown note type: (0x01000001) <----- HERE
> VMCOREINFO_XEN 0x00000fe4 Unknown note type: (0x00000000)
> $
Well there may be a couple issues with that approach. First, we need
a way to handle/recognize it on live systems. And second, I believe we'll need
a way to differentiate a pv_ops/xen kernel from a "dedicated" xen kernel.
And my guess is that the vmcore notes will not differentiate between the
two "flavors" of xen kernel (unless there's an entry in the VMCOREINFO_XEN
that will somehow differentiate the two).
Thank you for the good comment. I understand your plan, and I think it is
better than mine.
Thanks
Ken'ichi Ohmichi
5:39 a.m.
New subject: [PATCH 1/2] Add x86_64 linux-2.6.27 support: Catch up the PAGE_OFFSET change.
Ken'ichi Ohmichi wrote:
Hello Dave,
Dave Anderson wrote:
>>I think that a new virtualization_init() function is a good idea.
>>
>>How about checking whether "Xen" exists in PT_NOTE section of an vmcore
>>at virtualization_init()? I feel that it is safer than reading a xen_start_info
>>pointer.
>>
>>$ readelf -n vmcore
>>Notes at offset 0x00000120 with length 0x00001388:
>> Owner Data size Description
>> CORE 0x00000150 NT_PRSTATUS (prstatus structure)
>> Xen 0x00000020 Unknown note type: (0x01000002) <----- HERE
>> CORE 0x00000150 NT_PRSTATUS (prstatus structure)
>> Xen 0x00000020 Unknown note type: (0x01000002) <----- HERE
>> Xen 0x00000050 Unknown note type: (0x01000001) <----- HERE
>> VMCOREINFO_XEN 0x00000fe4 Unknown note type: (0x00000000)
>>$
>
>Well there may be a couple issues with that approach. First, we need
>a way to handle/recognize it on live systems. And second, I believe we'll need
>a way to differentiate a pv_ops/xen kernel from a "dedicated" xen kernel.
>And my guess is that the vmcore notes will not differentiate between the
>two "flavors" of xen kernel (unless there's an entry in the
VMCOREINFO_XEN
>that will somehow differentiate the two).
Thank you for the good comment. I understand your plan, and I think it is
better than mine.
Yeah, the more I look into it, the more it appears that support for
pv_ops/xen (as well as the other pv_ops clients) are going to require
significant changes.
For now, only "traditional xen" will be supported, and when pv_ops/xen
comes online, we'll take a look at how to merge support for it.
Here's my initial update to your patch -- it simply recognizes pv_ops
kernels, and disallows the setting of the ARCH_XEN in that case.
Dave
5957
days inactive
5960
days old
devel@lists.crash-utility.osci.io
4 comments
2 participants
participants (2)
-
Dave Anderson -
Ken'ichi Ohmichi