For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0"). http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd... http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7... This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

On 11/11/2014 06:22 AM, Laszlo Ersek wrote: > (Note: I'm not subscribed to either qemu-devel or the kexec list; please > keep me CC'd.) > > QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, > kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command. > > The resultant vmcore is usually analyzed with the "crash" utility. > > The original tool producing such files is kdump. Unlike the procedure > performed by QEMU, kdump runs from *within* the guest (under a kexec'd > kdump kernel), and has more information about the original guest kernel > state (which is being dumped) than QEMU. To QEMU, the guest kernel state > is opaque. > > For this reason, the kdump preparation logic in QEMU hardcodes a number > of fields in the kdump header. The direct issue is the "phys_base" > field. Refer to dump.c, functions create_header32(), create_header64(), > and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text > "0"). > > http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd... > > http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7... > > This works in most cases, because the guest Linux kernel indeed tends to > be loaded at guest-phys address 0. However, when the guest Linux kernel > is booted on top of OVMF (which has a somewhat unusual UEFI memory map), > then the guest Linux kernel is loaded at 16MB, thereby getting out of > sync with the phys_base=0 setting visible in the KDUMP header. > > This trips up the "crash" utility. > > Dave worked around the issue in "crash" for ELF format dumps -- "crash" > can identify QEMU as the originator of the vmcore by finding the QEMU > notes in the ELF vmcore. If those are present, then "crash" employs a > heuristic, probing for a phys_base up to 32MB, in 1MB steps. What advantages does KDUMP have over ELF?

On 11/12/2014 03:05 AM, Petr Tesarik wrote: > On Tue, 11 Nov 2014 12:27:44 -0500 > Christopher Covington <cov(a)codeaurora.org&gt; wrote: > >> On 11/11/2014 06:22 AM, Laszlo Ersek wrote: >>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please >>> keep me CC'd.) >>> >>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, >>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command. >>> >>> The resultant vmcore is usually analyzed with the "crash" utility. >>> >>> The original tool producing such files is kdump. Unlike the procedure >>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd >>> kdump kernel), and has more information about the original guest kernel >>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state >>> is opaque. >>> >>> For this reason, the kdump preparation logic in QEMU hardcodes a number >>> of fields in the kdump header. The direct issue is the "phys_base" >>> field. Refer to dump.c, functions create_header32(), create_header64(), >>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text >>> "0"). >>> >>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd... >>> >>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7... >>> >>> This works in most cases, because the guest Linux kernel indeed tends to >>> be loaded at guest-phys address 0. However, when the guest Linux kernel >>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map), >>> then the guest Linux kernel is loaded at 16MB, thereby getting out of >>> sync with the phys_base=0 setting visible in the KDUMP header. >>> >>> This trips up the "crash" utility. >>> >>> Dave worked around the issue in "crash" for ELF format dumps -- "crash" >>> can identify QEMU as the originator of the vmcore by finding the QEMU >>> notes in the ELF vmcore. If those are present, then "crash" employs a >>> heuristic, probing for a phys_base up to 32MB, in 1MB steps. >> >> What advantages does KDUMP have over ELF? > > It's smaller (data is compressed), and it contains a header with some > useful information (e.g. the crashed kernel's version and release). What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?

How does QEMU figure out the kernel version information?

On 11/12/2014 08:26 AM, Petr Tesarik wrote: > On Wed, 12 Nov 2014 08:18:04 -0500 > Christopher Covington <cov(a)codeaurora.org&gt; wrote: > >> On 11/12/2014 03:05 AM, Petr Tesarik wrote: >>> On Tue, 11 Nov 2014 12:27:44 -0500 >>> Christopher Covington <cov(a)codeaurora.org&gt; wrote: >>> >>>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote: >>>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please >>>>> keep me CC'd.) >>>>> >>>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, >>>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command. >>>>> >>>>> The resultant vmcore is usually analyzed with the "crash" utility. >>>>> >>>>> The original tool producing such files is kdump. Unlike the procedure >>>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd >>>>> kdump kernel), and has more information about the original guest kernel >>>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state >>>>> is opaque. >>>>> >>>>> For this reason, the kdump preparation logic in QEMU hardcodes a number >>>>> of fields in the kdump header. The direct issue is the "phys_base" >>>>> field. Refer to dump.c, functions create_header32(), create_header64(), >>>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text >>>>> "0"). >>>>> >>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd... >>>>> >>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7... >>>>> >>>>> This works in most cases, because the guest Linux kernel indeed tends to >>>>> be loaded at guest-phys address 0. However, when the guest Linux kernel >>>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map), >>>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of >>>>> sync with the phys_base=0 setting visible in the KDUMP header. >>>>> >>>>> This trips up the "crash" utility. >>>>> >>>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash" >>>>> can identify QEMU as the originator of the vmcore by finding the QEMU >>>>> notes in the ELF vmcore. If those are present, then "crash" employs a >>>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps. >>>> >>>> What advantages does KDUMP have over ELF? >>> >>> It's smaller (data is compressed), and it contains a header with some >>> useful information (e.g. the crashed kernel's version and release). >> >> What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz? > > Not the same thing. With KDUMP, each page is compressed separately, so > if a utility like crash needs a page from the middle, it can find it > and unpack it immediately. If we had an ELF.xz, then the whole file > must be unpacked before it can be used. And unpacking a few terabytes > takes ... a while. ;-) Understood on the ELF.xz approach, but why couldn't each page (or maybe a configurable size) be a SHF_COMPRESSED section?

On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <cov(a)codeaurora.org&gt; wrote: > On 11/12/2014 03:05 AM, Petr Tesarik wrote: >> On Tue, 11 Nov 2014 12:27:44 -0500 >> Christopher Covington <cov(a)codeaurora.org&gt; wrote: >> >>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote: >>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please >>>> keep me CC'd.) >>>> >>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, >>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command. >>>> >>>> The resultant vmcore is usually analyzed with the "crash" utility. >>>> >>>> The original tool producing such files is kdump. Unlike the procedure >>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd >>>> kdump kernel), and has more information about the original guest kernel >>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state >>>> is opaque. >>>> >>>> For this reason, the kdump preparation logic in QEMU hardcodes a number >>>> of fields in the kdump header. The direct issue is the "phys_base" >>>> field. Refer to dump.c, functions create_header32(), create_header64(), >>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text >>>> "0"). >>>> >>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd... >>>> >>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7... >>>> >>>> This works in most cases, because the guest Linux kernel indeed tends to >>>> be loaded at guest-phys address 0. However, when the guest Linux kernel >>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map), >>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of >>>> sync with the phys_base=0 setting visible in the KDUMP header. >>>> >>>> This trips up the "crash" utility. >>>> >>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash" >>>> can identify QEMU as the originator of the vmcore by finding the QEMU >>>> notes in the ELF vmcore. If those are present, then "crash" employs a >>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps. >>> >>> What advantages does KDUMP have over ELF? >> >> It's smaller (data is compressed), and it contains a header with some >> useful information (e.g. the crashed kernel's version and release).

> What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz? Not the same thing. With KDUMP, each page is compressed separately, so if a utility like crash needs a page from the middle, it can find it and unpack it immediately. If we had an ELF.xz, then the whole file must be unpacked before it can be used. And unpacking a few terabytes takes ... a while. ;-) > How does QEMU figure out the kernel version information? Good question. Who can answer this part?

Thanks Petr and Laszlo for entertaining my questions. I've got one last one if you have the time. On 11/12/2014 09:10 AM, Laszlo Ersek wrote: > On 11/12/14 14:26, Petr Tesarik wrote: >> On Wed, 12 Nov 2014 08:18:04 -0500 >> Christopher Covington <cov(a)codeaurora.org&gt; wrote: >> >>> On 11/12/2014 03:05 AM, Petr Tesarik wrote: >>>> On Tue, 11 Nov 2014 12:27:44 -0500 >>>> Christopher Covington <cov(a)codeaurora.org&gt; wrote: >>>> >>>>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote: >>>>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please >>>>>> keep me CC'd.) >>>>>> >>>>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, >>>>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command. >>>>>> >>>>>> The resultant vmcore is usually analyzed with the "crash" utility. >>>>>> >>>>>> The original tool producing such files is kdump. Unlike the procedure >>>>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd >>>>>> kdump kernel), and has more information about the original guest kernel >>>>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state >>>>>> is opaque. >>>>>> >>>>>> For this reason, the kdump preparation logic in QEMU hardcodes a number >>>>>> of fields in the kdump header. The direct issue is the "phys_base" >>>>>> field. Refer to dump.c, functions create_header32(), create_header64(), >>>>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text >>>>>> "0"). >>>>>> >>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd... >>>>>> >>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7... >>>>>> >>>>>> This works in most cases, because the guest Linux kernel indeed tends to >>>>>> be loaded at guest-phys address 0. However, when the guest Linux kernel >>>>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map), >>>>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of >>>>>> sync with the phys_base=0 setting visible in the KDUMP header. >>>>>> >>>>>> This trips up the "crash" utility. >>>>>> >>>>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash" >>>>>> can identify QEMU as the originator of the vmcore by finding the QEMU >>>>>> notes in the ELF vmcore. If those are present, then "crash" employs a >>>>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps. >>>>> >>>>> What advantages does KDUMP have over ELF? >>>> >>>> It's smaller (data is compressed), and it contains a header with some >>>> useful information (e.g. the crashed kernel's version and release). > > Another advantage is that all zero-filled pages are represented in the > kdump file by one shared zero page. > > The difference in speed of dumping is stunning. Would you expect using SHT_NOBITS to give a similar speedup to the ELF dumper?

On 11/12/2014 10:03 AM, Laszlo Ersek wrote: > On 11/12/14 15:48, Christopher Covington wrote: >> Thanks Petr and Laszlo for entertaining my questions. I've got one last one if >> you have the time. >> >> On 11/12/2014 09:10 AM, Laszlo Ersek wrote: >>> On 11/12/14 14:26, Petr Tesarik wrote: >>>> On Wed, 12 Nov 2014 08:18:04 -0500 >>>> Christopher Covington <cov(a)codeaurora.org&gt; wrote: >>>> >>>>> On 11/12/2014 03:05 AM, Petr Tesarik wrote: >>>>>> On Tue, 11 Nov 2014 12:27:44 -0500 >>>>>> Christopher Covington <cov(a)codeaurora.org&gt; wrote: >>>>>> >>>>>>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote: >>>>>>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please >>>>>>>> keep me CC'd.) >>>>>>>> >>>>>>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, >>>>>>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command. >>>>>>>> >>>>>>>> The resultant vmcore is usually analyzed with the "crash" utility. >>>>>>>> >>>>>>>> The original tool producing such files is kdump. Unlike the procedure >>>>>>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd >>>>>>>> kdump kernel), and has more information about the original guest kernel >>>>>>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state >>>>>>>> is opaque. >>>>>>>> >>>>>>>> For this reason, the kdump preparation logic in QEMU hardcodes a number >>>>>>>> of fields in the kdump header. The direct issue is the "phys_base" >>>>>>>> field. Refer to dump.c, functions create_header32(), create_header64(), >>>>>>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text >>>>>>>> "0"). >>>>>>>> >>>>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd... >>>>>>>> >>>>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7... >>>>>>>> >>>>>>>> This works in most cases, because the guest Linux kernel indeed tends to >>>>>>>> be loaded at guest-phys address 0. However, when the guest Linux kernel >>>>>>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map), >>>>>>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of >>>>>>>> sync with the phys_base=0 setting visible in the KDUMP header. >>>>>>>> >>>>>>>> This trips up the "crash" utility. >>>>>>>> >>>>>>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash" >>>>>>>> can identify QEMU as the originator of the vmcore by finding the QEMU >>>>>>>> notes in the ELF vmcore. If those are present, then "crash" employs a >>>>>>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps. >>>>>>> >>>>>>> What advantages does KDUMP have over ELF? >>>>>> >>>>>> It's smaller (data is compressed), and it contains a header with some >>>>>> useful information (e.g. the crashed kernel's version and release). >>> >>> Another advantage is that all zero-filled pages are represented in the >>> kdump file by one shared zero page. >>> >>> The difference in speed of dumping is stunning. >> >> Would you expect using SHT_NOBITS to give a similar speedup to the ELF dumper? > > Sorry, I don't know what SHT_NOBITS is. My newbie understanding is that SHT_NOBITS is the section type of the .bss section in an everyday executable.