8:09 a.m.
----- Original Message -----
From: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
To: ptesarik(a)suse.cz
Cc: lersek(a)redhat.com, kexec(a)lists.infradead.org
Subject: Re: uniquely identifying KDUMP files that originate from QEMU
Message-ID:
<20141112.120838.303682123986142686.d.hatayama(a)jp.fujitsu.com>
Content-Type: Text/Plain; charset=us-ascii
From: Petr Tesarik <ptesarik(a)suse.cz>
Subject: Re: uniquely identifying KDUMP files that originate from QEMU
Date: Tue, 11 Nov 2014 13:09:13 +0100
> On Tue, 11 Nov 2014 12:22:52 +0100
> Laszlo Ersek <lersek(a)redhat.com> wrote:
>
>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
>> keep me CC'd.)
>>
>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
>>
>> The resultant vmcore is usually analyzed with the "crash" utility.
>>
>> The original tool producing such files is kdump. Unlike the procedure
>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
>> kdump kernel), and has more information about the original guest kernel
>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
>> is opaque.
>>
>> For this reason, the kdump preparation logic in QEMU hardcodes a number
>> of fields in the kdump header. The direct issue is the "phys_base"
>> field. Refer to dump.c, functions create_header32(), create_header64(),
>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement
text
>> "0").
>>
>>
http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd...
>>
>>
http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7...
>>
>> This works in most cases, because the guest Linux kernel indeed tends to
>> be loaded at guest-phys address 0. However, when the guest Linux kernel
>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>> sync with the phys_base=0 setting visible in the KDUMP header.
>>
>> This trips up the "crash" utility.
>>
>> Dave worked around the issue in "crash" for ELF format dumps --
"crash"
>> can identify QEMU as the originator of the vmcore by finding the QEMU
>> notes in the ELF vmcore. If those are present, then "crash" employs a
>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
>>
>> Alas, the QEMU notes are not present in the KDUMP-format vmcores that
>> QEMU produces (they cannot be),
>
> Why? Since KDUMP format version 4, the complete ELF notes can be stored
> in the file (see offset_note, size_note fields in the sub-header).
>
Yes, the QEMU notes is present in kdump-compressed format. But
phys_base cannot be calculated only from qemu-side. We cannot do more
than the efforts crash utility does for workaround. So, the phys_base
value in kdump-sub header is now designed to have 0 now.
Anyway, phys_base is kernel information. To make it available for qemu
side, there's need to prepare a mechanism for qemu to have any access
to it.
One ad-hoc but simple way is to put phys_base value as part of
VMCOREINFO note information on kernel.
Although there has already been a similar one in VMCOREINFO, like
arch/x86/kernel/
==
void arch_crash_save_vmcoreinfo(void)
{
VMCOREINFO_SYMBOL(phys_base); <---- This
VMCOREINFO_SYMBOL(init_level4_pgt);
...
==
this is meangless, because this value is a virtual address assigned to
phys_base symbol. To refer to the value of phys_base itself, we need
the phys_base value we are about to get now.
So, instead, if we change this to save the value, not value of symbol
phys_base, we can get phys_base from the VMCOREINFO.
The VMCOREINFO consists simply of string. So it's easy to search
vmcore for it e.g. using strings and grep like this:
$ strings vmcore-3.10.0-121.el7.x86_64 | grep -E ".*VMCOREINFO.*" -A 100
VMCOREINFO
OSRELEASE=3.10.0-121.el7.x86_64
PAGESIZE=4096
...
SYMBOL(phys_base)=ffffffff818e5010 <-- though this is address of phys_base
now...
SYMBOL(init_level4_pgt)=ffffffff818de000
SYMBOL(node_data)=ffffffff819f1cc0
LENGTH(node_data)=1024
CRASHTIME=1399460394
...
This should also be useful to get phys_base of 2nd kernel, which is
inherently relocated kernel from a vmcore generated using qemu dump.
This is far from well-designed from qemu's point of view, but it would
be manually easier to get phys_base than now.
Obviously, the VMCOREINFO is available only if CONFIG_KEXEC is
enabled. Other users cannot use this.
--
Thanks.
HATAYAMA, Daisuke
I agree that the actual value of phys_base should be included in the vmcoreinfo.
However, it won't help in this case because the vmcoreinfo data is not
copied into the compressed dumpfile header. The offset_vmcoreinfo and
size_vmcoreinfo fields are zero.
Here's an example header dump of a QEMU-generated dumpfile:
crash> help -n
makedumpfile header:
signature: "makedumpfile"
type: 1
version: 1
all_flat_data:
num_array: 18695
array: 7f484b760010
file_size: 0
diskdump_data:
filename: vmcore.ovmf.rhel7.kdump-snappy
flags: c6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED|LZO_SUPPORTED|SNAPPY_SUPPORTED)
[FLAT]
dfd: 3
ofp: 3e441b1260
machine_type: 62 (EM_X86_64)
header: 1a68fe0
signature: "KDUMP "
header_version: 6
utsname:
sysname:
nodename:
release:
version:
machine: x86_64
domainname:
timestamp:
tv_sec: 0
tv_usec: 0
status: 4 (DUMP_DH_COMPRESSED_SNAPPY)
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 76
max_mapnr: 1245184
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 4
tasks[nr_cpus]: 0
0
0
0
sub_header: 0 (n/a)
sub_header_kdump: 1a69ff0
phys_base: 0
dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
split: 0
start_pfn: (unused)
end_pfn: (unused)
offset_vmcoreinfo: 0 (0x0)
size_vmcoreinfo: 0 (0x0)
offset_note: 4200 (0x1068)
size_note: 3232 (0xca0)
num_prstatus_notes: 4
notes_buf: 1a6b000
notes[0]: 1a6b000
notes[1]: 1a6b164
notes[2]: 1a6b2c8
notes[3]: 1a6b42c
NT_PRSTATUS_offset: 1068
11cc
1330
1494
offset_eraseinfo: 0 (0x0)
size_eraseinfo: 0 (0x0)
start_pfn_64: (unused)
end_pfn_64: (unused)
max_mapnr_64: 1245184 (0x130000)
data_offset: 4e000
block_size: 4096
block_shift: 12
bitmap: 7f484b713010
bitmap_len: 311296
max_mapnr: 1245184 (0x130000)
dumpable_bitmap: 7f484b6c6010
byte: 0
bit: 0
compressed_page: 1a8c660
curbufptr: 1a7f650
...
Note that QEMU does add self-generated register dumps above, but the special
"QEMU" note that is added to ELF kdumps is not included.
Also note that the kernel version information is also left zero-filled.
In any case, if either a QEMU note or a diskdump.data flag were added, I would
be more than happy.
Dave
9:01 a.m.
New subject: uniquely identifying KDUMP files that originate from QEMU
On 11/12/14 15:09, Dave Anderson wrote:
----- Original Message -----
> From: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
> To: ptesarik(a)suse.cz
> Cc: lersek(a)redhat.com, kexec(a)lists.infradead.org
> Subject: Re: uniquely identifying KDUMP files that originate from QEMU
> Message-ID:
> <20141112.120838.303682123986142686.d.hatayama(a)jp.fujitsu.com>
> Content-Type: Text/Plain; charset=us-ascii
>
> From: Petr Tesarik <ptesarik(a)suse.cz>
> Subject: Re: uniquely identifying KDUMP files that originate from QEMU
> Date: Tue, 11 Nov 2014 13:09:13 +0100
>
>> On Tue, 11 Nov 2014 12:22:52 +0100
>> Laszlo Ersek <lersek(a)redhat.com> wrote:
>>
>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
>>> keep me CC'd.)
>>>
>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
>>>
>>> The resultant vmcore is usually analyzed with the "crash" utility.
>>>
>>> The original tool producing such files is kdump. Unlike the procedure
>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
>>> kdump kernel), and has more information about the original guest kernel
>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
>>> is opaque.
>>>
>>> For this reason, the kdump preparation logic in QEMU hardcodes a number
>>> of fields in the kdump header. The direct issue is the "phys_base"
>>> field. Refer to dump.c, functions create_header32(), create_header64(),
>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement
text
>>> "0").
>>>
>>>
http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd...
>>>
>>>
http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7...
>>>
>>> This works in most cases, because the guest Linux kernel indeed tends to
>>> be loaded at guest-phys address 0. However, when the guest Linux kernel
>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>>> sync with the phys_base=0 setting visible in the KDUMP header.
>>>
>>> This trips up the "crash" utility.
>>>
>>> Dave worked around the issue in "crash" for ELF format dumps --
"crash"
>>> can identify QEMU as the originator of the vmcore by finding the QEMU
>>> notes in the ELF vmcore. If those are present, then "crash" employs
a
>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
>>>
>>> Alas, the QEMU notes are not present in the KDUMP-format vmcores that
>>> QEMU produces (they cannot be),
>>
>> Why? Since KDUMP format version 4, the complete ELF notes can be stored
>> in the file (see offset_note, size_note fields in the sub-header).
>>
>
> Yes, the QEMU notes is present in kdump-compressed format. But
> phys_base cannot be calculated only from qemu-side. We cannot do more
> than the efforts crash utility does for workaround. So, the phys_base
> value in kdump-sub header is now designed to have 0 now.
>
> Anyway, phys_base is kernel information. To make it available for qemu
> side, there's need to prepare a mechanism for qemu to have any access
> to it.
>
> One ad-hoc but simple way is to put phys_base value as part of
> VMCOREINFO note information on kernel.
>
> Although there has already been a similar one in VMCOREINFO, like
>
> arch/x86/kernel/
> ==
> void arch_crash_save_vmcoreinfo(void)
> {
> VMCOREINFO_SYMBOL(phys_base); <---- This
> VMCOREINFO_SYMBOL(init_level4_pgt);
>
> ...
> ==
>
> this is meangless, because this value is a virtual address assigned to
> phys_base symbol. To refer to the value of phys_base itself, we need
> the phys_base value we are about to get now.
>
> So, instead, if we change this to save the value, not value of symbol
> phys_base, we can get phys_base from the VMCOREINFO.
>
> The VMCOREINFO consists simply of string. So it's easy to search
> vmcore for it e.g. using strings and grep like this:
>
> $ strings vmcore-3.10.0-121.el7.x86_64 | grep -E ".*VMCOREINFO.*" -A 100
> VMCOREINFO
> OSRELEASE=3.10.0-121.el7.x86_64
> PAGESIZE=4096
> ...
> SYMBOL(phys_base)=ffffffff818e5010 <-- though this is address of phys_base
> now...
> SYMBOL(init_level4_pgt)=ffffffff818de000
> SYMBOL(node_data)=ffffffff819f1cc0
> LENGTH(node_data)=1024
> CRASHTIME=1399460394
> ...
>
> This should also be useful to get phys_base of 2nd kernel, which is
> inherently relocated kernel from a vmcore generated using qemu dump.
>
> This is far from well-designed from qemu's point of view, but it would
> be manually easier to get phys_base than now.
>
> Obviously, the VMCOREINFO is available only if CONFIG_KEXEC is
> enabled. Other users cannot use this.
>
> --
> Thanks.
> HATAYAMA, Daisuke
I agree that the actual value of phys_base should be included in the vmcoreinfo.
However, it won't help in this case because the vmcoreinfo data is not
copied into the compressed dumpfile header. The offset_vmcoreinfo and
size_vmcoreinfo fields are zero.
Here's an example header dump of a QEMU-generated dumpfile:
crash> help -n
makedumpfile header:
signature: "makedumpfile"
type: 1
version: 1
all_flat_data:
num_array: 18695
array: 7f484b760010
file_size: 0
diskdump_data:
filename: vmcore.ovmf.rhel7.kdump-snappy
flags: c6
(KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED|LZO_SUPPORTED|SNAPPY_SUPPORTED) [FLAT]
dfd: 3
ofp: 3e441b1260
machine_type: 62 (EM_X86_64)
header: 1a68fe0
signature: "KDUMP "
header_version: 6
utsname:
sysname:
nodename:
release:
version:
machine: x86_64
domainname:
timestamp:
tv_sec: 0
tv_usec: 0
status: 4 (DUMP_DH_COMPRESSED_SNAPPY)
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 76
max_mapnr: 1245184
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 4
tasks[nr_cpus]: 0
0
0
0
sub_header: 0 (n/a)
sub_header_kdump: 1a69ff0
phys_base: 0
dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
split: 0
start_pfn: (unused)
end_pfn: (unused)
offset_vmcoreinfo: 0 (0x0)
size_vmcoreinfo: 0 (0x0)
offset_note: 4200 (0x1068)
size_note: 3232 (0xca0)
num_prstatus_notes: 4
notes_buf: 1a6b000
notes[0]: 1a6b000
notes[1]: 1a6b164
notes[2]: 1a6b2c8
notes[3]: 1a6b42c
NT_PRSTATUS_offset: 1068
11cc
1330
1494
offset_eraseinfo: 0 (0x0)
size_eraseinfo: 0 (0x0)
start_pfn_64: (unused)
end_pfn_64: (unused)
max_mapnr_64: 1245184 (0x130000)
data_offset: 4e000
block_size: 4096
block_shift: 12
bitmap: 7f484b713010
bitmap_len: 311296
max_mapnr: 1245184 (0x130000)
dumpable_bitmap: 7f484b6c6010
byte: 0
bit: 0
compressed_page: 1a8c660
curbufptr: 1a7f650
...
Note that QEMU does add self-generated register dumps above, but the special
"QEMU" note that is added to ELF kdumps is not included.
Also note that the kernel version information is also left zero-filled.
In any case, if either a QEMU note or a diskdump.data flag were added, I would
be more than happy.
Looks like a new flag needs to be negotiated with many stake-holders,
but a QEMU note could be included even in the kdump format (not only the
ELF format) freely, and tools that don't recognize it would simply
ignore it. (And other tools that generate custom notes probably won't
clash with it.)
Is that correct? Because if it is, then (a) I didn't know it, (b) we
only need an agreement between "crash" and qemu.
Is the kdump format specified somewhere (as in, a PDF or text file)? I'd
like to look into this option if possible.
Also, is there a command line tool that dumps metadata from a kdump
file? (Quite like your "crash" invocation above, but I believe crash
won't even start without a matching symbol file.)
Thank you
Laszlo
9:45 a.m.
New subject: uniquely identifying KDUMP files that originate from QEMU
----- Original Message -----
On 11/12/14 15:09, Dave Anderson wrote:
>
>
> ----- Original Message -----
>> From: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
>> To: ptesarik(a)suse.cz
>> Cc: lersek(a)redhat.com, kexec(a)lists.infradead.org
>> Subject: Re: uniquely identifying KDUMP files that originate from QEMU
>> Message-ID:
>> <20141112.120838.303682123986142686.d.hatayama(a)jp.fujitsu.com>
>> Content-Type: Text/Plain; charset=us-ascii
>>
>> From: Petr Tesarik <ptesarik(a)suse.cz>
>> Subject: Re: uniquely identifying KDUMP files that originate from QEMU
>> Date: Tue, 11 Nov 2014 13:09:13 +0100
>>
>>> On Tue, 11 Nov 2014 12:22:52 +0100
>>> Laszlo Ersek <lersek(a)redhat.com> wrote:
>>>
>>>> (Note: I'm not subscribed to either qemu-devel or the kexec list;
please
>>>> keep me CC'd.)
>>>>
>>>> QEMU is able to dump the guest's memory in KDUMP format
(kdump-zlib,
>>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP
command.
>>>>
>>>> The resultant vmcore is usually analyzed with the "crash"
utility.
>>>>
>>>> The original tool producing such files is kdump. Unlike the procedure
>>>> performed by QEMU, kdump runs from *within* the guest (under a
kexec'd
>>>> kdump kernel), and has more information about the original guest kernel
>>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel
state
>>>> is opaque.
>>>>
>>>> For this reason, the kdump preparation logic in QEMU hardcodes a number
>>>> of fields in the kdump header. The direct issue is the
"phys_base"
>>>> field. Refer to dump.c, functions create_header32(), create_header64(),
>>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the
replacement text
>>>> "0").
>>>>
>>>>
http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd...
>>>>
>>>>
http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7...
>>>>
>>>> This works in most cases, because the guest Linux kernel indeed tends
to
>>>> be loaded at guest-phys address 0. However, when the guest Linux kernel
>>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory
map),
>>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>>>> sync with the phys_base=0 setting visible in the KDUMP header.
>>>>
>>>> This trips up the "crash" utility.
>>>>
>>>> Dave worked around the issue in "crash" for ELF format dumps
-- "crash"
>>>> can identify QEMU as the originator of the vmcore by finding the QEMU
>>>> notes in the ELF vmcore. If those are present, then "crash"
employs a
>>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
>>>>
>>>> Alas, the QEMU notes are not present in the KDUMP-format vmcores that
>>>> QEMU produces (they cannot be),
>>>
>>> Why? Since KDUMP format version 4, the complete ELF notes can be stored
>>> in the file (see offset_note, size_note fields in the sub-header).
>>>
>>
>> Yes, the QEMU notes is present in kdump-compressed format. But
>> phys_base cannot be calculated only from qemu-side. We cannot do more
>> than the efforts crash utility does for workaround. So, the phys_base
>> value in kdump-sub header is now designed to have 0 now.
>>
>> Anyway, phys_base is kernel information. To make it available for qemu
>> side, there's need to prepare a mechanism for qemu to have any access
>> to it.
>>
>> One ad-hoc but simple way is to put phys_base value as part of
>> VMCOREINFO note information on kernel.
>>
>> Although there has already been a similar one in VMCOREINFO, like
>>
>> arch/x86/kernel/
>> ==
>> void arch_crash_save_vmcoreinfo(void)
>> {
>> VMCOREINFO_SYMBOL(phys_base); <---- This
>> VMCOREINFO_SYMBOL(init_level4_pgt);
>>
>> ...
>> ==
>>
>> this is meangless, because this value is a virtual address assigned to
>> phys_base symbol. To refer to the value of phys_base itself, we need
>> the phys_base value we are about to get now.
>>
>> So, instead, if we change this to save the value, not value of symbol
>> phys_base, we can get phys_base from the VMCOREINFO.
>>
>> The VMCOREINFO consists simply of string. So it's easy to search
>> vmcore for it e.g. using strings and grep like this:
>>
>> $ strings vmcore-3.10.0-121.el7.x86_64 | grep -E ".*VMCOREINFO.*" -A
100
>> VMCOREINFO
>> OSRELEASE=3.10.0-121.el7.x86_64
>> PAGESIZE=4096
>> ...
>> SYMBOL(phys_base)=ffffffff818e5010 <-- though this is address of
>> phys_base
>> now...
>> SYMBOL(init_level4_pgt)=ffffffff818de000
>> SYMBOL(node_data)=ffffffff819f1cc0
>> LENGTH(node_data)=1024
>> CRASHTIME=1399460394
>> ...
>>
>> This should also be useful to get phys_base of 2nd kernel, which is
>> inherently relocated kernel from a vmcore generated using qemu dump.
>>
>> This is far from well-designed from qemu's point of view, but it would
>> be manually easier to get phys_base than now.
>>
>> Obviously, the VMCOREINFO is available only if CONFIG_KEXEC is
>> enabled. Other users cannot use this.
>>
>> --
>> Thanks.
>> HATAYAMA, Daisuke
>
> I agree that the actual value of phys_base should be included in the
> vmcoreinfo.
>
> However, it won't help in this case because the vmcoreinfo data is not
> copied into the compressed dumpfile header. The offset_vmcoreinfo and
> size_vmcoreinfo fields are zero.
>
> Here's an example header dump of a QEMU-generated dumpfile:
>
> crash> help -n
> makedumpfile header:
> signature: "makedumpfile"
> type: 1
> version: 1
> all_flat_data:
> num_array: 18695
> array: 7f484b760010
> file_size: 0
>
> diskdump_data:
> filename: vmcore.ovmf.rhel7.kdump-snappy
> flags: c6
> (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED|LZO_SUPPORTED|SNAPPY_SUPPORTED)
> [FLAT]
> dfd: 3
> ofp: 3e441b1260
> machine_type: 62 (EM_X86_64)
>
> header: 1a68fe0
> signature: "KDUMP "
> header_version: 6
> utsname:
> sysname:
> nodename:
> release:
> version:
> machine: x86_64
> domainname:
> timestamp:
> tv_sec: 0
> tv_usec: 0
> status: 4 (DUMP_DH_COMPRESSED_SNAPPY)
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 76
> max_mapnr: 1245184
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 4
> tasks[nr_cpus]: 0
> 0
> 0
> 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 1a69ff0
> phys_base: 0
> dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
> split: 0
> start_pfn: (unused)
> end_pfn: (unused)
> offset_vmcoreinfo: 0 (0x0)
> size_vmcoreinfo: 0 (0x0)
> offset_note: 4200 (0x1068)
> size_note: 3232 (0xca0)
> num_prstatus_notes: 4
> notes_buf: 1a6b000
> notes[0]: 1a6b000
> notes[1]: 1a6b164
> notes[2]: 1a6b2c8
> notes[3]: 1a6b42c
> NT_PRSTATUS_offset: 1068
> 11cc
> 1330
> 1494
> offset_eraseinfo: 0 (0x0)
> size_eraseinfo: 0 (0x0)
> start_pfn_64: (unused)
> end_pfn_64: (unused)
> max_mapnr_64: 1245184 (0x130000)
>
> data_offset: 4e000
> block_size: 4096
> block_shift: 12
> bitmap: 7f484b713010
> bitmap_len: 311296
> max_mapnr: 1245184 (0x130000)
> dumpable_bitmap: 7f484b6c6010
> byte: 0
> bit: 0
> compressed_page: 1a8c660
> curbufptr: 1a7f650
> ...
>
> Note that QEMU does add self-generated register dumps above, but the special
> "QEMU" note that is added to ELF kdumps is not included.
>
> Also note that the kernel version information is also left zero-filled.
>
> In any case, if either a QEMU note or a diskdump.data flag were added, I would
> be more than happy.
Looks like a new flag needs to be negotiated with many stake-holders,
but a QEMU note could be included even in the kdump format (not only the
ELF format) freely, and tools that don't recognize it would simply
ignore it. (And other tools that generate custom notes probably won't
clash with it.)
Is that correct? Because if it is, then (a) I didn't know it, (b) we
only need an agreement between "crash" and qemu.
Agreed.
Is the kdump format specified somewhere (as in, a PDF or text file)?
I'd
like to look into this option if possible.
I don't know. Bernhard Walle, formerly of SUSE, used to keep a text file
stored, but all the links are dead now:
[Crash-utility] Re: Kdump compressed format
https://www.redhat.com/archives/crash-utility/2008-August/msg00014.html
But his does certainly wouldn't have anything w/respect to QEMU notes.
Maybe Petr or the Fujitsu guys have a pointer?
But anyway, as it turns out, in QEMU ELF kdumps create one of the special
"QEMU" notes for each cpu:
$ readelf --notes vmcore
Notes at offset 0x000001c8 with length 0x00000ca0:
Owner Data size Description
CORE 0x00000150 NT_PRSTATUS (prstatus structure)
CORE 0x00000150 NT_PRSTATUS (prstatus structure)
CORE 0x00000150 NT_PRSTATUS (prstatus structure)
CORE 0x00000150 NT_PRSTATUS (prstatus structure)
QEMU 0x000001b0 Unknown note type: (0x00000000)
QEMU 0x000001b0 Unknown note type: (0x00000000)
QEMU 0x000001b0 Unknown note type: (0x00000000)
QEMU 0x000001b0 Unknown note type: (0x00000000)
$
Here are the contents of each QEMU note:
crash> help -n
... [ cut ] ...
Elf64_Nhdr:
n_namesz: 5 ("QEMU")
n_descsz: 432
n_type: 0 (?)
000001b000000001 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000001
ffffffff81dd5228 ffffffff81a01ec8
ffffffff81a01ec8 0000000000000000
0000000000000000 00000013911d5f29
0000000000000000 ffffffff81c00480
0000000000000000 ffffffffffffffff
000000000309f000 ffffffff810375ab
0000000000000246 ffffffff00000010
0000000000a09b00 0000000000000000
ffffffff00000018 0000000000c09300
0000000000000000 ffffffff00000018
0000000000c09300 0000000000000000
ffffffff00000000 0000000000000000
0000000000000000 ffffffff00000000
0000000000000000 ffff880003200000
ffffffff00000018 0000000000c09300
0000000000000000 ffffffff00000000
0000000000000000 0000000000000000
0000208700000040 0000000000008b00
ffff880003213b40 0000007f00000000
0000000000000000 ffff880003204000
00000fff00000000 0000000000000000
ffffffff81dd2000 000000008005003b
0000000000000000 0000000001b2e000
0000000007b18000 00000000000006f0
Elf64_Nhdr:
n_namesz: 5 ("QEMU")
n_descsz: 432
n_type: 0 (?)
000001b000000001 ffffffff81a93760
000000000000000c 0000000080802001
0000000000000000 00000000000000ff
00000000000000f0 ffff880002287e88
ffff880002287e88 ffff880002287e50
ffff880002287e54 000009149661fc2b
ffff88001e6abe78 ffff880002287ef8
00000000fffffffe 0000000000000000
ffffffff81bfed40 ffffffff810375ba
0000000000000002 ffffffff00000010
0000000000a09b00 0000000000000000
ffffffff00000018 0000000000c09300
0000000000000000 ffffffff00000018
0000000000c09300 0000000000000000
ffffffff00000000 0000000000000000
0000000000000000 ffffffff00000000
0000000000000000 ffff880002280000
ffffffff00000018 0000000000c09300
0000000000000000 ffffffff00000000
0000000000000000 0000000000000000
0000208700000040 0000000000008b00
ffff880002293b40 0000007f00000000
0000000000000000 ffff880002284000
00000fff00000000 0000000000000000
ffffffff81dd2000 000000008005003b
0000000000000000 0000000002162570
000000001aab8000 00000000000006e0
Elf64_Nhdr:
n_namesz: 5 ("QEMU")
n_descsz: 432
n_type: 0 (?)
000001b000000001 ffffffff81a93760
000000000000000c 0000000080802001
0000000000000000 00000000000000ff
00000000000000f0 ffff880002307e88
ffff880002307e88 ffff880002307e50
ffff880002307e54 000009143aed494c
ffff88001e6dfe78 ffff880002307ef8
00000000fffffffe 0000000000000000
ffffffff81bfed40 ffffffff810375ba
0000000000000002 ffffffff00000010
0000000000a09b00 0000000000000000
ffffffff00000018 0000000000c09300
0000000000000000 ffffffff00000018
0000000000c09300 0000000000000000
ffffffff00000000 0000000000000000
0000000000000000 ffffffff00000000
0000000000000000 ffff880002300000
ffffffff00000018 0000000000c09300
0000000000000000 ffffffff00000000
0000000000000000 0000000000000000
0000208700000040 0000000000008b00
ffff880002313b40 0000007f00000000
0000000000000000 ffff880002304000
00000fff00000000 0000000000000000
ffffffff81dd2000 000000008005003b
0000000000000000 00007fd1a029c000
000000001d5c7000 00000000000006e0
Elf64_Nhdr:
n_namesz: 5 ("QEMU")
n_descsz: 432
n_type: 0 (?)
000001b000000001 ffffffff81a93760
000000000000000c 0000000080802001
0000000000000000 00000000000000ff
00000000000000f0 ffff880002387e88
ffff880002387e88 ffff880002387e50
ffff880002387e54 0000091497285969
0000000000000000 ffff880002387ef8
00000000fffffffe 0000000000000000
ffffffff81bfed40 ffffffff810375ba
0000000000000046 ffffffff00000010
0000000000a09b00 0000000000000000
ffffffff00000018 0000000000c09300
0000000000000000 ffffffff00000018
0000000000c09300 0000000000000000
ffffffff00000000 0000000000000000
0000000000000000 ffffffff00000000
0000000000000000 ffff880002380000
ffffffff00000018 0000000000c09300
0000000000000000 ffffffff00000000
0000000000000000 0000000000000000
0000208700000040 0000000000008b00
ffff880002393b40 0000007f00000000
0000000000000000 ffff880002384000
00000fff00000000 0000000000000000
ffffffff81dd2000 000000008005003b
0000000000000000 00007f981fe51000
000000001f214000 00000000000006e0
crash>
I'm not sure what the data consists of. The crash utility simply checks
for the existance of a note with a "QEMU" name string.
Also, is there a command line tool that dumps metadata from a kdump
file? (Quite like your "crash" invocation above, but I believe crash
won't even start without a matching symbol file.)
I don't know of any, although Petr mentioned something about a "kdumpid"
tool?
It's on sourceforge, but I've never heard of it until today, and don't know
if it dumps the full contents of headers.
However, you can get the header dump I showed before without a vmlinux file
by using the -d debug flag on the vmcore:
$ crash -d1 vmcore.ovmf.rhel7.kdump-zlib
crash 7.0.8
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
vmcore.ovmf.rhel7.kdump-zlib: FLAT
compressed kdump: header->utsname.machine: x86_64
makedumpfile header:
signature: "makedumpfile"
type: 1
version: 1
all_flat_data:
num_array: 13851
array: 7f3c0a0da010
file_size: 0
diskdump_data:
filename: vmcore.ovmf.rhel7.kdump-zlib
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED) [FLAT]
dfd: 3
ofp: 0
machine_type: 62 (EM_X86_64)
header: 1c9bfe0
signature: "KDUMP "
header_version: 6
utsname:
sysname:
nodename:
release:
version:
machine: x86_64
domainname:
timestamp:
tv_sec: 0
tv_usec: 0
status: 1 (DUMP_DH_COMPRESSED_ZLIB)
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 76
max_mapnr: 1245184
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 4
tasks[nr_cpus]: 0
0
0
0
sub_header: 0 (n/a)
sub_header_kdump: 1c9cff0
phys_base: 0
dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
split: 0
start_pfn: (unused)
end_pfn: (unused)
offset_vmcoreinfo: 0 (0x0)
size_vmcoreinfo: 0 (0x0)
offset_note: 4200 (0x1068)
size_note: 3232 (0xca0)
num_prstatus_notes: 4
notes_buf: 1c9e000
notes[0]: 1c9e000
notes[1]: 1c9e164
notes[2]: 1c9e2c8
notes[3]: 1c9e42c
NT_PRSTATUS_offset: 1068
11cc
1330
1494
offset_eraseinfo: 0 (0x0)
size_eraseinfo: 0 (0x0)
start_pfn_64: (unused)
end_pfn_64: (unused)
max_mapnr_64: 1245184 (0x130000)
data_offset: 4e000
block_size: 4096
block_shift: 12
bitmap: 7f3c0a08d010
bitmap_len: 311296
max_mapnr: 1245184 (0x130000)
dumpable_bitmap: 7f3c0a040010
byte: 0
bit: 0
compressed_page: 1cbf660
curbufptr: 0
page_cache_hdr[0]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1caf650
pg_hit_count: 0
page_cache_hdr[1]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cb0650
pg_hit_count: 0
page_cache_hdr[2]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cb1650
pg_hit_count: 0
page_cache_hdr[3]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cb2650
pg_hit_count: 0
page_cache_hdr[4]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cb3650
pg_hit_count: 0
page_cache_hdr[5]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cb4650
pg_hit_count: 0
page_cache_hdr[6]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cb5650
pg_hit_count: 0
page_cache_hdr[7]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cb6650
pg_hit_count: 0
page_cache_hdr[8]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cb7650
pg_hit_count: 0
page_cache_hdr[9]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cb8650
pg_hit_count: 0
page_cache_hdr[10]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cb9650
pg_hit_count: 0
page_cache_hdr[11]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cba650
pg_hit_count: 0
page_cache_hdr[12]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cbb650
pg_hit_count: 0
page_cache_hdr[13]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cbc650
pg_hit_count: 0
page_cache_hdr[14]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cbd650
pg_hit_count: 0
page_cache_hdr[15]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 1cbe650
pg_hit_count: 0
page_cache_buf: 1caf650
evict_index: 0
evictions: 0
accesses: 0
cached_reads: 0
valid_pages: 1caecc0
crash: namelist argument required
Usage:
crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
Enter "crash -h" for details.
$
Dave
7:08 p.m.
New subject: uniquely identifying KDUMP files that originate from QEMU
From: Dave Anderson <anderson(a)redhat.com>
Subject: Re: uniquely identifying KDUMP files that originate from QEMU
Date: Wed, 12 Nov 2014 09:09:34 -0500
----- Original Message -----
> From: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
> To: ptesarik(a)suse.cz
> Cc: lersek(a)redhat.com, kexec(a)lists.infradead.org
> Subject: Re: uniquely identifying KDUMP files that originate from QEMU
> Message-ID:
> <20141112.120838.303682123986142686.d.hatayama(a)jp.fujitsu.com>
> Content-Type: Text/Plain; charset=us-ascii
>
> From: Petr Tesarik <ptesarik(a)suse.cz>
> Subject: Re: uniquely identifying KDUMP files that originate from QEMU
> Date: Tue, 11 Nov 2014 13:09:13 +0100
>
> > On Tue, 11 Nov 2014 12:22:52 +0100
> > Laszlo Ersek <lersek(a)redhat.com> wrote:
> >
> >> (Note: I'm not subscribed to either qemu-devel or the kexec list;
please
> >> keep me CC'd.)
> >>
> >> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
> >> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP
command.
> >>
> >> The resultant vmcore is usually analyzed with the "crash"
utility.
> >>
> >> The original tool producing such files is kdump. Unlike the procedure
> >> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
> >> kdump kernel), and has more information about the original guest kernel
> >> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
> >> is opaque.
> >>
> >> For this reason, the kdump preparation logic in QEMU hardcodes a number
> >> of fields in the kdump header. The direct issue is the
"phys_base"
> >> field. Refer to dump.c, functions create_header32(), create_header64(),
> >> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement
text
> >> "0").
> >>
> >>
http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd...
> >>
> >>
http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7...
> >>
> >> This works in most cases, because the guest Linux kernel indeed tends to
> >> be loaded at guest-phys address 0. However, when the guest Linux kernel
> >> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
> >> then the guest Linux kernel is loaded at 16MB, thereby getting out of
> >> sync with the phys_base=0 setting visible in the KDUMP header.
> >>
> >> This trips up the "crash" utility.
> >>
> >> Dave worked around the issue in "crash" for ELF format dumps --
"crash"
> >> can identify QEMU as the originator of the vmcore by finding the QEMU
> >> notes in the ELF vmcore. If those are present, then "crash"
employs a
> >> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
> >>
> >> Alas, the QEMU notes are not present in the KDUMP-format vmcores that
> >> QEMU produces (they cannot be),
> >
> > Why? Since KDUMP format version 4, the complete ELF notes can be stored
> > in the file (see offset_note, size_note fields in the sub-header).
> >
>
> Yes, the QEMU notes is present in kdump-compressed format. But
> phys_base cannot be calculated only from qemu-side. We cannot do more
> than the efforts crash utility does for workaround. So, the phys_base
> value in kdump-sub header is now designed to have 0 now.
>
> Anyway, phys_base is kernel information. To make it available for qemu
> side, there's need to prepare a mechanism for qemu to have any access
> to it.
>
> One ad-hoc but simple way is to put phys_base value as part of
> VMCOREINFO note information on kernel.
>
> Although there has already been a similar one in VMCOREINFO, like
>
> arch/x86/kernel/
> ==
> void arch_crash_save_vmcoreinfo(void)
> {
> VMCOREINFO_SYMBOL(phys_base); <---- This
> VMCOREINFO_SYMBOL(init_level4_pgt);
>
> ...
> ==
>
> this is meangless, because this value is a virtual address assigned to
> phys_base symbol. To refer to the value of phys_base itself, we need
> the phys_base value we are about to get now.
>
> So, instead, if we change this to save the value, not value of symbol
> phys_base, we can get phys_base from the VMCOREINFO.
>
> The VMCOREINFO consists simply of string. So it's easy to search
> vmcore for it e.g. using strings and grep like this:
>
> $ strings vmcore-3.10.0-121.el7.x86_64 | grep -E ".*VMCOREINFO.*" -A 100
> VMCOREINFO
> OSRELEASE=3.10.0-121.el7.x86_64
> PAGESIZE=4096
> ...
> SYMBOL(phys_base)=ffffffff818e5010 <-- though this is address of phys_base
> now...
> SYMBOL(init_level4_pgt)=ffffffff818de000
> SYMBOL(node_data)=ffffffff819f1cc0
> LENGTH(node_data)=1024
> CRASHTIME=1399460394
> ...
>
> This should also be useful to get phys_base of 2nd kernel, which is
> inherently relocated kernel from a vmcore generated using qemu dump.
>
> This is far from well-designed from qemu's point of view, but it would
> be manually easier to get phys_base than now.
>
> Obviously, the VMCOREINFO is available only if CONFIG_KEXEC is
> enabled. Other users cannot use this.
>
> --
> Thanks.
> HATAYAMA, Daisuke
I agree that the actual value of phys_base should be included in the vmcoreinfo.
However, it won't help in this case because the vmcoreinfo data is not
copied into the compressed dumpfile header. The offset_vmcoreinfo and
size_vmcoreinfo fields are zero.
Yes, so I said:
> This is far from well-designed from qemu's point of view, but
it would
> be manually easier to get phys_base than now.
This is just an ad-hoc way.
Here's an example header dump of a QEMU-generated dumpfile:
crash> help -n
makedumpfile header:
signature: "makedumpfile"
type: 1
version: 1
all_flat_data:
num_array: 18695
array: 7f484b760010
file_size: 0
diskdump_data:
filename: vmcore.ovmf.rhel7.kdump-snappy
flags: c6
(KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED|LZO_SUPPORTED|SNAPPY_SUPPORTED) [FLAT]
dfd: 3
ofp: 3e441b1260
machine_type: 62 (EM_X86_64)
header: 1a68fe0
signature: "KDUMP "
header_version: 6
utsname:
sysname:
nodename:
release:
version:
machine: x86_64
domainname:
timestamp:
tv_sec: 0
tv_usec: 0
status: 4 (DUMP_DH_COMPRESSED_SNAPPY)
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 76
max_mapnr: 1245184
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 4
tasks[nr_cpus]: 0
0
0
0
sub_header: 0 (n/a)
sub_header_kdump: 1a69ff0
phys_base: 0
dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
split: 0
start_pfn: (unused)
end_pfn: (unused)
offset_vmcoreinfo: 0 (0x0)
size_vmcoreinfo: 0 (0x0)
offset_note: 4200 (0x1068)
size_note: 3232 (0xca0)
num_prstatus_notes: 4
notes_buf: 1a6b000
notes[0]: 1a6b000
notes[1]: 1a6b164
notes[2]: 1a6b2c8
notes[3]: 1a6b42c
NT_PRSTATUS_offset: 1068
11cc
1330
1494
offset_eraseinfo: 0 (0x0)
size_eraseinfo: 0 (0x0)
start_pfn_64: (unused)
end_pfn_64: (unused)
max_mapnr_64: 1245184 (0x130000)
data_offset: 4e000
block_size: 4096
block_shift: 12
bitmap: 7f484b713010
bitmap_len: 311296
max_mapnr: 1245184 (0x130000)
dumpable_bitmap: 7f484b6c6010
byte: 0
bit: 0
compressed_page: 1a8c660
curbufptr: 1a7f650
...
Note that QEMU does add self-generated register dumps above, but the special
"QEMU" note that is added to ELF kdumps is not included.
Sorry, I didn't know this, and there's no reason not to add it.
Also note that the kernel version information is also left
zero-filled.
This is what I intended. Retrieving data from vmcore should be done in
crash utility or makedumpfile.
In any case, if either a QEMU note or a diskdump.data flag were
added, I would
be more than happy.
Dave
The absence of QEMU note is different from my intension. This is
regression agast ELF. We must add it.
--
Thanks.
HATAYAMA, Daisuke
9:21 a.m.
New subject: uniquely identifying KDUMP files that originate from QEMU
----- Original Message -----
From: Dave Anderson <anderson(a)redhat.com>
Subject: Re: uniquely identifying KDUMP files that originate from QEMU
Date: Wed, 12 Nov 2014 09:09:34 -0500
>
>
> ----- Original Message -----
>> From: HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com>
>> To: ptesarik(a)suse.cz
>> Cc: lersek(a)redhat.com, kexec(a)lists.infradead.org
>> Subject: Re: uniquely identifying KDUMP files that originate from QEMU
>> Message-ID:
>> <20141112.120838.303682123986142686.d.hatayama(a)jp.fujitsu.com>
>> Content-Type: Text/Plain; charset=us-ascii
>>
>> From: Petr Tesarik <ptesarik(a)suse.cz>
>> Subject: Re: uniquely identifying KDUMP files that originate from QEMU
>> Date: Tue, 11 Nov 2014 13:09:13 +0100
>>
>> > On Tue, 11 Nov 2014 12:22:52 +0100
>> > Laszlo Ersek <lersek(a)redhat.com> wrote:
>> >
>> >> (Note: I'm not subscribed to either qemu-devel or the kexec list;
>> >> please
>> >> keep me CC'd.)
>> >>
>> >> QEMU is able to dump the guest's memory in KDUMP format
(kdump-zlib,
>> >> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP
command.
>> >>
>> >> The resultant vmcore is usually analyzed with the "crash"
utility.
>> >>
>> >> The original tool producing such files is kdump. Unlike the procedure
>> >> performed by QEMU, kdump runs from *within* the guest (under a
kexec'd
>> >> kdump kernel), and has more information about the original guest
kernel
>> >> state (which is being dumped) than QEMU. To QEMU, the guest kernel
>> >> state
>> >> is opaque.
>> >>
>> >> For this reason, the kdump preparation logic in QEMU hardcodes a
number
>> >> of fields in the kdump header. The direct issue is the
"phys_base"
>> >> field. Refer to dump.c, functions create_header32(),
create_header64(),
>> >> and "include/sysemu/dump.h", macro PHYS_BASE (with the
replacement text
>> >> "0").
>> >>
>> >>
http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd...
>> >>
>> >>
http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7...
>> >>
>> >> This works in most cases, because the guest Linux kernel indeed tends
>> >> to
>> >> be loaded at guest-phys address 0. However, when the guest Linux
kernel
>> >> is booted on top of OVMF (which has a somewhat unusual UEFI memory
>> >> map),
>> >> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>> >> sync with the phys_base=0 setting visible in the KDUMP header.
>> >>
>> >> This trips up the "crash" utility.
>> >>
>> >> Dave worked around the issue in "crash" for ELF format dumps
-- "crash"
>> >> can identify QEMU as the originator of the vmcore by finding the QEMU
>> >> notes in the ELF vmcore. If those are present, then "crash"
employs a
>> >> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
>> >>
>> >> Alas, the QEMU notes are not present in the KDUMP-format vmcores that
>> >> QEMU produces (they cannot be),
>> >
>> > Why? Since KDUMP format version 4, the complete ELF notes can be stored
>> > in the file (see offset_note, size_note fields in the sub-header).
>> >
>>
>> Yes, the QEMU notes is present in kdump-compressed format. But
>> phys_base cannot be calculated only from qemu-side. We cannot do more
>> than the efforts crash utility does for workaround. So, the phys_base
>> value in kdump-sub header is now designed to have 0 now.
>>
>> Anyway, phys_base is kernel information. To make it available for qemu
>> side, there's need to prepare a mechanism for qemu to have any access
>> to it.
>>
>> One ad-hoc but simple way is to put phys_base value as part of
>> VMCOREINFO note information on kernel.
>>
>> Although there has already been a similar one in VMCOREINFO, like
>>
>> arch/x86/kernel/
>> ==
>> void arch_crash_save_vmcoreinfo(void)
>> {
>> VMCOREINFO_SYMBOL(phys_base); <---- This
>> VMCOREINFO_SYMBOL(init_level4_pgt);
>>
>> ...
>> ==
>>
>> this is meangless, because this value is a virtual address assigned to
>> phys_base symbol. To refer to the value of phys_base itself, we need
>> the phys_base value we are about to get now.
>>
>> So, instead, if we change this to save the value, not value of symbol
>> phys_base, we can get phys_base from the VMCOREINFO.
>>
>> The VMCOREINFO consists simply of string. So it's easy to search
>> vmcore for it e.g. using strings and grep like this:
>>
>> $ strings vmcore-3.10.0-121.el7.x86_64 | grep -E ".*VMCOREINFO.*" -A
100
>> VMCOREINFO
>> OSRELEASE=3.10.0-121.el7.x86_64
>> PAGESIZE=4096
>> ...
>> SYMBOL(phys_base)=ffffffff818e5010 <-- though this is address of
>> phys_base
>> now...
>> SYMBOL(init_level4_pgt)=ffffffff818de000
>> SYMBOL(node_data)=ffffffff819f1cc0
>> LENGTH(node_data)=1024
>> CRASHTIME=1399460394
>> ...
>>
>> This should also be useful to get phys_base of 2nd kernel, which is
>> inherently relocated kernel from a vmcore generated using qemu dump.
>>
>> This is far from well-designed from qemu's point of view, but it would
>> be manually easier to get phys_base than now.
>>
>> Obviously, the VMCOREINFO is available only if CONFIG_KEXEC is
>> enabled. Other users cannot use this.
>>
>> --
>> Thanks.
>> HATAYAMA, Daisuke
>
> I agree that the actual value of phys_base should be included in the
> vmcoreinfo.
>
> However, it won't help in this case because the vmcoreinfo data is not
> copied into the compressed dumpfile header. The offset_vmcoreinfo and
> size_vmcoreinfo fields are zero.
Yes, so I said:
>> This is far from well-designed from qemu's point of view, but it would
>> be manually easier to get phys_base than now.
This is just an ad-hoc way.
>
> Here's an example header dump of a QEMU-generated dumpfile:
>
> crash> help -n
> makedumpfile header:
> signature: "makedumpfile"
> type: 1
> version: 1
> all_flat_data:
> num_array: 18695
> array: 7f484b760010
> file_size: 0
>
> diskdump_data:
> filename: vmcore.ovmf.rhel7.kdump-snappy
> flags: c6
> (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED|LZO_SUPPORTED|SNAPPY_SUPPORTED)
> [FLAT]
> dfd: 3
> ofp: 3e441b1260
> machine_type: 62 (EM_X86_64)
>
> header: 1a68fe0
> signature: "KDUMP "
> header_version: 6
> utsname:
> sysname:
> nodename:
> release:
> version:
> machine: x86_64
> domainname:
> timestamp:
> tv_sec: 0
> tv_usec: 0
> status: 4 (DUMP_DH_COMPRESSED_SNAPPY)
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 76
> max_mapnr: 1245184
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 4
> tasks[nr_cpus]: 0
> 0
> 0
> 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 1a69ff0
> phys_base: 0
> dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
> split: 0
> start_pfn: (unused)
> end_pfn: (unused)
> offset_vmcoreinfo: 0 (0x0)
> size_vmcoreinfo: 0 (0x0)
> offset_note: 4200 (0x1068)
> size_note: 3232 (0xca0)
> num_prstatus_notes: 4
> notes_buf: 1a6b000
> notes[0]: 1a6b000
> notes[1]: 1a6b164
> notes[2]: 1a6b2c8
> notes[3]: 1a6b42c
> NT_PRSTATUS_offset: 1068
> 11cc
> 1330
> 1494
> offset_eraseinfo: 0 (0x0)
> size_eraseinfo: 0 (0x0)
> start_pfn_64: (unused)
> end_pfn_64: (unused)
> max_mapnr_64: 1245184 (0x130000)
>
> data_offset: 4e000
> block_size: 4096
> block_shift: 12
> bitmap: 7f484b713010
> bitmap_len: 311296
> max_mapnr: 1245184 (0x130000)
> dumpable_bitmap: 7f484b6c6010
> byte: 0
> bit: 0
> compressed_page: 1a8c660
> curbufptr: 1a7f650
> ...
>
> Note that QEMU does add self-generated register dumps above, but the
> special
> "QEMU" note that is added to ELF kdumps is not included.
>
Sorry, I didn't know this, and there's no reason not to add it.
> Also note that the kernel version information is also left zero-filled.
>
This is what I intended. Retrieving data from vmcore should be done in
crash utility or makedumpfile.
> In any case, if either a QEMU note or a diskdump.data flag were added, I would
> be more than happy.
>
> Dave
The absence of QEMU note is different from my intension. This is
regression agast ELF. We must add it.
Not necessary -- as it turns out, the QEMU notes are located in the compressed
kdump notes section following the NT_PRSTATUS notes:
http://lists.infradead.org/pipermail/kexec/2014-November/012974.html
It's just that the notes-gathering code in the crash utility was only
looking for and storing NT_PRSTATUS note information.
Thanks,
Dave
3666
days inactive
3667
days old
devel@lists.crash-utility.osci.io
4 comments
3 participants
participants (3)
-
Dave Anderson -
HATAYAMA Daisuke -
Laszlo Ersek