9:23 a.m.
ccing kexec list, vmcore-dmesg also uses vmcoreinfo related to printk..
-----Original Message-----
----- Original Message -----
> Hello Dave,
>
> You may or may not be aware that we are working on replacing [0] the
> Linux printk ringbuffer. Rather than a single buffer containing a single
> struct type, the new ringbuffer makes use of several different structs.
Yes, I am most definitely aware...
>
> I am writing to ask your advice about how this should be exported for
> the crash utility. Should all struct sizes and field offsets be
> exported? It would look something like this:
>
> VMCOREINFO_SYMBOL(prb);
>
> VMCOREINFO_STRUCT_SIZE(printk_ringbuffer);
> VMCOREINFO_OFFSET(printk_ringbuffer, desc_ring);
> VMCOREINFO_OFFSET(printk_ringbuffer, text_data_ring);
> VMCOREINFO_OFFSET(printk_ringbuffer, dict_data_ring);
> VMCOREINFO_OFFSET(printk_ringbuffer, fail);
>
> VMCOREINFO_STRUCT_SIZE(prb_desc_ring);
> VMCOREINFO_OFFSET(prb_desc_ring, count_bits);
> VMCOREINFO_OFFSET(prb_desc_ring, descs);
> VMCOREINFO_OFFSET(prb_desc_ring, head_id);
> VMCOREINFO_OFFSET(prb_desc_ring, tail_id);
>
> VMCOREINFO_STRUCT_SIZE(prb_desc);
> VMCOREINFO_OFFSET(prb_desc, info);
> VMCOREINFO_OFFSET(prb_desc, state_var);
> VMCOREINFO_OFFSET(prb_desc, text_blk_lpos);
> VMCOREINFO_OFFSET(prb_desc, dict_blk_lpos);
>
> VMCOREINFO_STRUCT_SIZE(prb_data_blk_lpos);
> VMCOREINFO_OFFSET(prb_data_blk_lpos, begin);
> VMCOREINFO_OFFSET(prb_data_blk_lpos, next);
>
> VMCOREINFO_STRUCT_SIZE(printk_info);
> VMCOREINFO_OFFSET(printk_info, seq);
> VMCOREINFO_OFFSET(printk_info, ts_nsec);
> VMCOREINFO_OFFSET(printk_info, text_len);
> VMCOREINFO_OFFSET(printk_info, dict_len);
> VMCOREINFO_OFFSET(printk_info, caller_id);
>
> VMCOREINFO_STRUCT_SIZE(prb_data_ring);
> VMCOREINFO_OFFSET(prb_data_ring, size_bits);
> VMCOREINFO_OFFSET(prb_data_ring, data);
> VMCOREINFO_OFFSET(prb_data_ring, head_id);
> VMCOREINFO_OFFSET(prb_data_ring, tail_id);
>
> Or would it be enough to just recognize the new "prb" symbol and have
> all the structures defined in the crash utility? If the latter is
> preferred, should some sort of version number be exported? Or is the
> kernel version number enough?
first I don't think we can depend on the kernel version because distribution
kernels backport upstream patches. So we will need a version number of the
ringbuffer if we choose that way.
I think that "exporting all things" can sometimes reflect changes in kernel
automatically and can reduce tool side effort more than "exporting a version
number". But the former requires a lot of entries and it might be hard for
us to track the changes. So having an explicit version might be better and
I'm ok with the latter.
But I hope no missing update of the version number..
Any thoughts from vmcore-dmesg side?
Thanks,
Kazu
>
> I appreciate your feedback.
>
> John Ogness
With respect to the crash utility, there are two answers.
When running crash session normally, i.e. running "crash vmlinux vmcore", the
runtime
"log" command does not use any VMCOREINFO entries that happen to be attached to
a dumpfile.
Since crash has the vmlinux debuginfo data available, it uses its own interfaces to get
all kernel symbol and structure related information.
But there is a little-used capability where the the vmlinux file is not required,
but rather just the vmcore, in its "crash --log vmcore" feature. That
functionality
does require the VMCOREINFO entries to extract/dump the log, and exit. Honestly I wish
I had never even introduced that feature. And I wonder if it were deprecated,
would anybody care?
However, your question is highly relevant to the makedumpfile(8) facility
for its "makedumpfile --dump-dmesg" option. Since it doesn't have the
luxury of a vmlinux file, it needs all of the VMCOREINF_xxx items. Kazuhito
Hagio is the makedumpfile maintainer, and since he is the primary customer
of the VMCOREINFO entries, he would be a better person to answer your
question.
That being said, due the sheer number VMCOREINFO entries required, I like
your idea of providing a single version number. But I defer to Kazu for
his preference.
Thanks,
Dave
9:43 a.m.
New subject: new printk ringbuffer interface
----- Original Message -----
ccing kexec list, vmcore-dmesg also uses vmcoreinfo related to
printk..
> -----Original Message-----
>
> ----- Original Message -----
> > Hello Dave,
> >
> > You may or may not be aware that we are working on replacing [0] the
> > Linux printk ringbuffer. Rather than a single buffer containing a single
> > struct type, the new ringbuffer makes use of several different structs.
>
> Yes, I am most definitely aware...
>
> >
> > I am writing to ask your advice about how this should be exported for
> > the crash utility. Should all struct sizes and field offsets be
> > exported? It would look something like this:
> >
> > VMCOREINFO_SYMBOL(prb);
> >
> > VMCOREINFO_STRUCT_SIZE(printk_ringbuffer);
> > VMCOREINFO_OFFSET(printk_ringbuffer, desc_ring);
> > VMCOREINFO_OFFSET(printk_ringbuffer, text_data_ring);
> > VMCOREINFO_OFFSET(printk_ringbuffer, dict_data_ring);
> > VMCOREINFO_OFFSET(printk_ringbuffer, fail);
> >
> > VMCOREINFO_STRUCT_SIZE(prb_desc_ring);
> > VMCOREINFO_OFFSET(prb_desc_ring, count_bits);
> > VMCOREINFO_OFFSET(prb_desc_ring, descs);
> > VMCOREINFO_OFFSET(prb_desc_ring, head_id);
> > VMCOREINFO_OFFSET(prb_desc_ring, tail_id);
> >
> > VMCOREINFO_STRUCT_SIZE(prb_desc);
> > VMCOREINFO_OFFSET(prb_desc, info);
> > VMCOREINFO_OFFSET(prb_desc, state_var);
> > VMCOREINFO_OFFSET(prb_desc, text_blk_lpos);
> > VMCOREINFO_OFFSET(prb_desc, dict_blk_lpos);
> >
> > VMCOREINFO_STRUCT_SIZE(prb_data_blk_lpos);
> > VMCOREINFO_OFFSET(prb_data_blk_lpos, begin);
> > VMCOREINFO_OFFSET(prb_data_blk_lpos, next);
> >
> > VMCOREINFO_STRUCT_SIZE(printk_info);
> > VMCOREINFO_OFFSET(printk_info, seq);
> > VMCOREINFO_OFFSET(printk_info, ts_nsec);
> > VMCOREINFO_OFFSET(printk_info, text_len);
> > VMCOREINFO_OFFSET(printk_info, dict_len);
> > VMCOREINFO_OFFSET(printk_info, caller_id);
> >
> > VMCOREINFO_STRUCT_SIZE(prb_data_ring);
> > VMCOREINFO_OFFSET(prb_data_ring, size_bits);
> > VMCOREINFO_OFFSET(prb_data_ring, data);
> > VMCOREINFO_OFFSET(prb_data_ring, head_id);
> > VMCOREINFO_OFFSET(prb_data_ring, tail_id);
> >
> > Or would it be enough to just recognize the new "prb" symbol and
have
> > all the structures defined in the crash utility? If the latter is
> > preferred, should some sort of version number be exported? Or is the
> > kernel version number enough?
first I don't think we can depend on the kernel version because distribution
kernels backport upstream patches. So we will need a version number of the
ringbuffer if we choose that way.
With respect to the kernel version, you are absolutely correct, as we've been
burnt by that before with backports to distribution and the upstream longterm
kernels. But I think John was talking about exporting a printk-structure-set
version number, so I think we're on the same page.
Also, if we go the version-number route, there would still not be a requirement
for the crash utility to duplicate the kernel data structures in its sources.
As John's proof-of-concept patch showed, it can still use the traditional
manner of getting structure sizes and member offsets. With the version number
exported, there may have to be a few small adjustments in the MEMBER_OFFSET_INIT()
calls, but it would be fairly straight-forward to maintain.
But of course makedumpfile would have to replicate the kernel data structures.
Thanks,
Dave
10:23 a.m.
New subject: new printk ringbuffer interface
-----Original Message-----
----- Original Message -----
> ccing kexec list, vmcore-dmesg also uses vmcoreinfo related to printk..
>
> > -----Original Message-----
> >
> > ----- Original Message -----
> > > Hello Dave,
> > >
> > > You may or may not be aware that we are working on replacing [0] the
> > > Linux printk ringbuffer. Rather than a single buffer containing a single
> > > struct type, the new ringbuffer makes use of several different structs.
> >
> > Yes, I am most definitely aware...
> >
> > >
> > > I am writing to ask your advice about how this should be exported for
> > > the crash utility. Should all struct sizes and field offsets be
> > > exported? It would look something like this:
> > >
> > > VMCOREINFO_SYMBOL(prb);
> > >
> > > VMCOREINFO_STRUCT_SIZE(printk_ringbuffer);
> > > VMCOREINFO_OFFSET(printk_ringbuffer, desc_ring);
> > > VMCOREINFO_OFFSET(printk_ringbuffer, text_data_ring);
> > > VMCOREINFO_OFFSET(printk_ringbuffer, dict_data_ring);
> > > VMCOREINFO_OFFSET(printk_ringbuffer, fail);
> > >
> > > VMCOREINFO_STRUCT_SIZE(prb_desc_ring);
> > > VMCOREINFO_OFFSET(prb_desc_ring, count_bits);
> > > VMCOREINFO_OFFSET(prb_desc_ring, descs);
> > > VMCOREINFO_OFFSET(prb_desc_ring, head_id);
> > > VMCOREINFO_OFFSET(prb_desc_ring, tail_id);
> > >
> > > VMCOREINFO_STRUCT_SIZE(prb_desc);
> > > VMCOREINFO_OFFSET(prb_desc, info);
> > > VMCOREINFO_OFFSET(prb_desc, state_var);
> > > VMCOREINFO_OFFSET(prb_desc, text_blk_lpos);
> > > VMCOREINFO_OFFSET(prb_desc, dict_blk_lpos);
> > >
> > > VMCOREINFO_STRUCT_SIZE(prb_data_blk_lpos);
> > > VMCOREINFO_OFFSET(prb_data_blk_lpos, begin);
> > > VMCOREINFO_OFFSET(prb_data_blk_lpos, next);
> > >
> > > VMCOREINFO_STRUCT_SIZE(printk_info);
> > > VMCOREINFO_OFFSET(printk_info, seq);
> > > VMCOREINFO_OFFSET(printk_info, ts_nsec);
> > > VMCOREINFO_OFFSET(printk_info, text_len);
> > > VMCOREINFO_OFFSET(printk_info, dict_len);
> > > VMCOREINFO_OFFSET(printk_info, caller_id);
> > >
> > > VMCOREINFO_STRUCT_SIZE(prb_data_ring);
> > > VMCOREINFO_OFFSET(prb_data_ring, size_bits);
> > > VMCOREINFO_OFFSET(prb_data_ring, data);
> > > VMCOREINFO_OFFSET(prb_data_ring, head_id);
> > > VMCOREINFO_OFFSET(prb_data_ring, tail_id);
> > >
> > > Or would it be enough to just recognize the new "prb" symbol and
have
> > > all the structures defined in the crash utility? If the latter is
> > > preferred, should some sort of version number be exported? Or is the
> > > kernel version number enough?
>
> first I don't think we can depend on the kernel version because distribution
> kernels backport upstream patches. So we will need a version number of the
> ringbuffer if we choose that way.
With respect to the kernel version, you are absolutely correct, as we've been
burnt by that before with backports to distribution and the upstream longterm
kernels. But I think John was talking about exporting a printk-structure-set
version number, so I think we're on the same page.
Also, if we go the version-number route, there would still not be a requirement
for the crash utility to duplicate the kernel data structures in its sources.
As John's proof-of-concept patch showed, it can still use the traditional
manner of getting structure sizes and member offsets. With the version number
exported, there may have to be a few small adjustments in the MEMBER_OFFSET_INIT()
calls, but it would be fairly straight-forward to maintain.
But of course makedumpfile would have to replicate the kernel data structures.
Ah, indeed. I vaguely thought makedumpfile also might be able to do like crash
by setting sizes and offsets for each version.. But it will be hard to set them
manually for every architectures. I'll think about it more.
Thanks,
Kazu
2:21 p.m.
New subject: new printk ringbuffer interface
-----Original Message-----
> -----Original Message-----
> ----- Original Message -----
> > ccing kexec list, vmcore-dmesg also uses vmcoreinfo related to printk..
> >
> > > -----Original Message-----
> > >
> > > ----- Original Message -----
> > > > Hello Dave,
> > > >
> > > > You may or may not be aware that we are working on replacing [0] the
> > > > Linux printk ringbuffer. Rather than a single buffer containing a
single
> > > > struct type, the new ringbuffer makes use of several different
structs.
> > >
> > > Yes, I am most definitely aware...
> > >
> > > >
> > > > I am writing to ask your advice about how this should be exported
for
> > > > the crash utility. Should all struct sizes and field offsets be
> > > > exported? It would look something like this:
> > > >
> > > > VMCOREINFO_SYMBOL(prb);
> > > >
> > > > VMCOREINFO_STRUCT_SIZE(printk_ringbuffer);
> > > > VMCOREINFO_OFFSET(printk_ringbuffer, desc_ring);
> > > > VMCOREINFO_OFFSET(printk_ringbuffer, text_data_ring);
> > > > VMCOREINFO_OFFSET(printk_ringbuffer, dict_data_ring);
> > > > VMCOREINFO_OFFSET(printk_ringbuffer, fail);
> > > >
> > > > VMCOREINFO_STRUCT_SIZE(prb_desc_ring);
> > > > VMCOREINFO_OFFSET(prb_desc_ring, count_bits);
> > > > VMCOREINFO_OFFSET(prb_desc_ring, descs);
> > > > VMCOREINFO_OFFSET(prb_desc_ring, head_id);
> > > > VMCOREINFO_OFFSET(prb_desc_ring, tail_id);
> > > >
> > > > VMCOREINFO_STRUCT_SIZE(prb_desc);
> > > > VMCOREINFO_OFFSET(prb_desc, info);
> > > > VMCOREINFO_OFFSET(prb_desc, state_var);
> > > > VMCOREINFO_OFFSET(prb_desc, text_blk_lpos);
> > > > VMCOREINFO_OFFSET(prb_desc, dict_blk_lpos);
> > > >
> > > > VMCOREINFO_STRUCT_SIZE(prb_data_blk_lpos);
> > > > VMCOREINFO_OFFSET(prb_data_blk_lpos, begin);
> > > > VMCOREINFO_OFFSET(prb_data_blk_lpos, next);
> > > >
> > > > VMCOREINFO_STRUCT_SIZE(printk_info);
> > > > VMCOREINFO_OFFSET(printk_info, seq);
> > > > VMCOREINFO_OFFSET(printk_info, ts_nsec);
> > > > VMCOREINFO_OFFSET(printk_info, text_len);
> > > > VMCOREINFO_OFFSET(printk_info, dict_len);
> > > > VMCOREINFO_OFFSET(printk_info, caller_id);
> > > >
> > > > VMCOREINFO_STRUCT_SIZE(prb_data_ring);
> > > > VMCOREINFO_OFFSET(prb_data_ring, size_bits);
> > > > VMCOREINFO_OFFSET(prb_data_ring, data);
> > > > VMCOREINFO_OFFSET(prb_data_ring, head_id);
> > > > VMCOREINFO_OFFSET(prb_data_ring, tail_id);
> > > >
> > > > Or would it be enough to just recognize the new "prb"
symbol and have
> > > > all the structures defined in the crash utility? If the latter is
> > > > preferred, should some sort of version number be exported? Or is the
> > > > kernel version number enough?
> >
> > first I don't think we can depend on the kernel version because
distribution
> > kernels backport upstream patches. So we will need a version number of the
> > ringbuffer if we choose that way.
>
> With respect to the kernel version, you are absolutely correct, as we've been
> burnt by that before with backports to distribution and the upstream longterm
> kernels. But I think John was talking about exporting a printk-structure-set
> version number, so I think we're on the same page.
>
> Also, if we go the version-number route, there would still not be a requirement
> for the crash utility to duplicate the kernel data structures in its sources.
> As John's proof-of-concept patch showed, it can still use the traditional
> manner of getting structure sizes and member offsets. With the version number
> exported, there may have to be a few small adjustments in the MEMBER_OFFSET_INIT()
> calls, but it would be fairly straight-forward to maintain.
>
> But of course makedumpfile would have to replicate the kernel data structures.
Ah, indeed. I vaguely thought makedumpfile also might be able to do like crash
by setting sizes and offsets for each version.. But it will be hard to set them
manually for every architectures. I'll think about it more.
makedumpfile has some architecture or kernel version specific values in it and
switches them by conditions, but the reason why we can do this is that they are
mostly constants that rarely change and can be got from the kernel source easily.
With respect to structure sizes and memger offsets, I don't think of a way to do
this easily.
And then if we define all the structures in makedumpfile with the version number,
how can we implement them and switch for each version? The new ringbuffer is
structured very much, I cannot imagine that it can be handled efficiently.
Any ideas?
If there is no efficient way, we will need all of the entries in VMCOREINFO.
Thanks,
Kazu
7:53 a.m.
New subject: new printk ringbuffer interface
On 2020-04-23, HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab(a)nec.com> wrote:
> Should all struct sizes and field offsets be exported? It
> would look something like this:
>
> VMCOREINFO_SYMBOL(prb);
>
> VMCOREINFO_STRUCT_SIZE(printk_ringbuffer);
> VMCOREINFO_OFFSET(printk_ringbuffer, desc_ring);
> VMCOREINFO_OFFSET(printk_ringbuffer, text_data_ring);
> VMCOREINFO_OFFSET(printk_ringbuffer, dict_data_ring);
> VMCOREINFO_OFFSET(printk_ringbuffer, fail);
>
> VMCOREINFO_STRUCT_SIZE(prb_desc_ring);
> VMCOREINFO_OFFSET(prb_desc_ring, count_bits);
> VMCOREINFO_OFFSET(prb_desc_ring, descs);
> VMCOREINFO_OFFSET(prb_desc_ring, head_id);
> VMCOREINFO_OFFSET(prb_desc_ring, tail_id);
>
> VMCOREINFO_STRUCT_SIZE(prb_desc);
> VMCOREINFO_OFFSET(prb_desc, info);
> VMCOREINFO_OFFSET(prb_desc, state_var);
> VMCOREINFO_OFFSET(prb_desc, text_blk_lpos);
> VMCOREINFO_OFFSET(prb_desc, dict_blk_lpos);
>
> VMCOREINFO_STRUCT_SIZE(prb_data_blk_lpos);
> VMCOREINFO_OFFSET(prb_data_blk_lpos, begin);
> VMCOREINFO_OFFSET(prb_data_blk_lpos, next);
>
> VMCOREINFO_STRUCT_SIZE(printk_info);
> VMCOREINFO_OFFSET(printk_info, seq);
> VMCOREINFO_OFFSET(printk_info, ts_nsec);
> VMCOREINFO_OFFSET(printk_info, text_len);
> VMCOREINFO_OFFSET(printk_info, dict_len);
> VMCOREINFO_OFFSET(printk_info, caller_id);
>
> VMCOREINFO_STRUCT_SIZE(prb_data_ring);
> VMCOREINFO_OFFSET(prb_data_ring, size_bits);
> VMCOREINFO_OFFSET(prb_data_ring, data);
> VMCOREINFO_OFFSET(prb_data_ring, head_id);
> VMCOREINFO_OFFSET(prb_data_ring, tail_id);
If there is no efficient way, we will need all of the entries in
VMCOREINFO.
It seems like a lot to export everything, but I don't have a problem
with it. If we decide to export everything (which I expect we will need
to do), then I would change my crash(8) implementation to also rely only
on the VMCOREINFO. I see no point in having some implementations using
debug data and other implementations using VMCOREINFO data, if
VMCOREINFO has everything that is needed.
John Ogness
8:16 a.m.
New subject: new printk ringbuffer interface
----- Original Message -----
On 2020-04-23, HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab(a)nec.com>
wrote:
>> Should all struct sizes and field offsets be exported? It
>> would look something like this:
>>
>> VMCOREINFO_SYMBOL(prb);
>>
>> VMCOREINFO_STRUCT_SIZE(printk_ringbuffer);
>> VMCOREINFO_OFFSET(printk_ringbuffer, desc_ring);
>> VMCOREINFO_OFFSET(printk_ringbuffer, text_data_ring);
>> VMCOREINFO_OFFSET(printk_ringbuffer, dict_data_ring);
>> VMCOREINFO_OFFSET(printk_ringbuffer, fail);
>>
>> VMCOREINFO_STRUCT_SIZE(prb_desc_ring);
>> VMCOREINFO_OFFSET(prb_desc_ring, count_bits);
>> VMCOREINFO_OFFSET(prb_desc_ring, descs);
>> VMCOREINFO_OFFSET(prb_desc_ring, head_id);
>> VMCOREINFO_OFFSET(prb_desc_ring, tail_id);
>>
>> VMCOREINFO_STRUCT_SIZE(prb_desc);
>> VMCOREINFO_OFFSET(prb_desc, info);
>> VMCOREINFO_OFFSET(prb_desc, state_var);
>> VMCOREINFO_OFFSET(prb_desc, text_blk_lpos);
>> VMCOREINFO_OFFSET(prb_desc, dict_blk_lpos);
>>
>> VMCOREINFO_STRUCT_SIZE(prb_data_blk_lpos);
>> VMCOREINFO_OFFSET(prb_data_blk_lpos, begin);
>> VMCOREINFO_OFFSET(prb_data_blk_lpos, next);
>>
>> VMCOREINFO_STRUCT_SIZE(printk_info);
>> VMCOREINFO_OFFSET(printk_info, seq);
>> VMCOREINFO_OFFSET(printk_info, ts_nsec);
>> VMCOREINFO_OFFSET(printk_info, text_len);
>> VMCOREINFO_OFFSET(printk_info, dict_len);
>> VMCOREINFO_OFFSET(printk_info, caller_id);
>>
>> VMCOREINFO_STRUCT_SIZE(prb_data_ring);
>> VMCOREINFO_OFFSET(prb_data_ring, size_bits);
>> VMCOREINFO_OFFSET(prb_data_ring, data);
>> VMCOREINFO_OFFSET(prb_data_ring, head_id);
>> VMCOREINFO_OFFSET(prb_data_ring, tail_id);
>
> If there is no efficient way, we will need all of the entries in
> VMCOREINFO.
It seems like a lot to export everything, but I don't have a problem
with it. If we decide to export everything (which I expect we will need
to do), then I would change my crash(8) implementation to also rely only
on the VMCOREINFO. I see no point in having some implementations using
debug data and other implementations using VMCOREINFO data, if
VMCOREINFO has everything that is needed.
Please don't -- the crash utility supports ~15 different dumpfile
formats, the majority of which do *not* contain VMCOREINFO data.
For that reason, I try to avoid using VMCOREINFO data whenever possible,
precisely because the relevant data can be gathered from the vmlinux symbol
table and debuginfo data.
Dave
12:18 a.m.
New subject: Crash issues with ARM aarch64 on a live system
Apologies if this the wrong place to ask this question. If in fact this is the wrong alias
to post this question, appreciate a pointer in the right direction. We have been trying to
get crash working on a live kernel (aarch64) but have not been able to.
Based on the crash help text:
--machdep option=value
Pass an option and value pair to machine-dependent code. These
architecture-specific option/pairs should only be required in
very rare circumstances:
ARM64:
phys_offset=<physical-address>
kimage_voffset=<kimage_voffset-value>
max_physmem_bits=<value>
vabits_actual=<value>
We probably need to specify the proper values for phys_offset and kimage_voffset.
kimage_offset is in /proc/kallsyms, but not phys_offset. It is just not clear where the
values should come from. I did try using the value of kimage_offset found in
/proc/kallsyms but that did not help. It is not clear where one needs to get the values
from (and in fact a casual reading of the crash source seems to try to compute the values
automatically)
Here is some additional information:
# uname -a
Linux bringup 5.1.0 #1 SMP Fri Oct 11 17:01:55 UTC 2019 aarch64 GNU/Linux
# crash -s
crash: /proc/kcore: No such file or directory
WARNING: could not find MAGIC_START!
WARNING: cannot read linux_banner string
crash: /boot/vmlinux-5.1.0 and /dev/mem do not match!
This is with additional debugging turned on:
# ./crash --version
crash 7.2.8++
Copyright (C) 2002-2020 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu".
Here is crash running with -d flag:
find_booted_kernel: check: /boot/vmlinux-5.1.0
find_booted_kernel: found: /boot/vmlinux-5.1.0
get_live_memory_source: /dev/mem
/proc/version:
Linux version 5.1.0 (oe-user@oe-host) (gcc version 7.3.0 (GCC)) #1 SMP Fri Oct 11 17:01:55
UTC 2019
/boot/vmlinux-5.1.0:
Linux version 5.1.0 (oe-user@oe-host) (gcc version 7.3.0 (GCC)) #1 SMP Fri Oct 11 17:01:55
UTC 2019
crash: /proc/kcore: No such file or directory
readmem: read_dev_mem() -> /dev/mem
VA_BITS: 48
kimage_voffset: fffeffff88000000
phys_offset: 0
physvirt_offset: 800000000000
gdb --quiet /boot/vmlinux-5.1.0
GETBUF(328 -> 0)
GETBUF(1500 -> 1)
FREEBUF(1)
FREEBUF(0)
<readmem: ffff0000109c63b8, KVADDR, "kernel_config_data", 32768, (ROE),
aaaad82d4630>
<read_dev_mem: addr: ffff0000109c63b8 paddr: 889c63b8 cnt: 3144>
<read_dev_mem: addr: ffff0000109c7000 paddr: 889c7000 cnt: 4096>
<read_dev_mem: addr: ffff0000109c8000 paddr: 889c8000 cnt: 4096>
<read_dev_mem: addr: ffff0000109c9000 paddr: 889c9000 cnt: 4096>
<read_dev_mem: addr: ffff0000109ca000 paddr: 889ca000 cnt: 4096>
<read_dev_mem: addr: ffff0000109cb000 paddr: 889cb000 cnt: 4096>
<read_dev_mem: addr: ffff0000109cc000 paddr: 889cc000 cnt: 4096>
<read_dev_mem: addr: ffff0000109cd000 paddr: 889cd000 cnt: 4096>
<read_dev_mem: addr: ffff0000109ce000 paddr: 889ce000 cnt: 952>
<readmem: ffff0000109c63b0, KVADDR, "kernel_config_data MAGIC_START", 8,
(ROE), ffffe6a6d5a0>
<read_dev_mem: addr: ffff0000109c63b0 paddr: 889c63b0 cnt: 8>
WARNING: could not find MAGIC_START!
GETBUF(328 -> 0)
FREEBUF(0)
GETBUF(8 -> 0)
<readmem: ffff000010e1c6d8, KVADDR, "possible", 8, (ROE|Q), aaaacd15f258>
<read_dev_mem: addr: ffff000010e1c6d8 paddr: 88e1c6d8 cnt: 8>
cpu_possible_mask: cpus: (none)
<readmem: ffff000010e1c6d0, KVADDR, "present", 8, (ROE|Q), aaaacd15f258>
<read_dev_mem: addr: ffff000010e1c6d0 paddr: 88e1c6d0 cnt: 8>
cpu_present_mask: cpus: (none)
<readmem: ffff000010e1c6c8, KVADDR, "online", 8, (ROE|Q), aaaacd15f258>
<read_dev_mem: addr: ffff000010e1c6c8 paddr: 88e1c6c8 cnt: 8>
cpu_online_mask: cpus: (none)
<readmem: ffff000010e1c6e0, KVADDR, "active", 8, (ROE|Q), aaaacd15f258>
<read_dev_mem: addr: ffff000010e1c6e0 paddr: 88e1c6e0 cnt: 8>
cpu_active_mask: cpus: (none)
FREEBUF(0)
GETBUF(328 -> 0)
FREEBUF(0)
GETBUF(328 -> 0)
FREEBUF(0)
GETBUF(328 -> 0)
FREEBUF(0)
<readmem: ffff000010ef8150, KVADDR, "shadow_timekeeper xtime_sec", 8, (ROE),
ffffe6a650b8>
<read_dev_mem: addr: ffff000010ef8150 paddr: 88ef8150 cnt: 8>
xtime timespec.tv_sec: 0: Wed Dec 31 16:00:00 1969
<readmem: ffff000010e2533c, KVADDR, "init_uts_ns", 390, (ROE),
aaaacd0fd194>
<read_dev_mem: addr: ffff000010e2533c paddr: 88e2533c cnt: 390>
utsname:
sysname:
nodename:
release:
version:
machine:
domainname:
base kernel version: 0.0.0
<readmem: ffff0000109b0098, KVADDR, "accessible check", 8, (ROE|Q),
ffffe6a64770>
<read_dev_mem: addr: ffff0000109b0098 paddr: 889b0098 cnt: 8>
<readmem: ffff0000109b0098, KVADDR, "read_string characters", 1499, (ROE|Q),
ffffe6a64ae8>
<read_dev_mem: addr: ffff0000109b0098 paddr: 889b0098 cnt: 1499>
WARNING: cannot read linux_banner string
/proc/version:
Linux version 5.1.0 (oe-user@oe-host) (gcc version 7.3.0 (GCC)) #1 SMP Fri Oct 11 17:01:55
UTC 2019
linux_banner:
crash: /boot/vmlinux-5.1.0 and /dev/mem do not match!
1:18 a.m.
New subject: Crash issues with ARM aarch64 on a live system
On Wed, May 20, 2020 at 10:19 PM Siamak Nazari <siamaknazari(a)yahoo.com> wrote:
Apologies if this the wrong place to ask this question. If in fact this is the wrong
alias to post this question, appreciate a pointer in the right direction. We have been
trying to get crash working on a live kernel (aarch64) but have not been able to.
Based on the crash help text:
--machdep option=value
Pass an option and value pair to machine-dependent code. These
architecture-specific option/pairs should only be required in
very rare circumstances:
ARM64:
phys_offset=<physical-address>
kimage_voffset=<kimage_voffset-value>
max_physmem_bits=<value>
vabits_actual=<value>
We probably need to specify the proper values for phys_offset and kimage_voffset.
kimage_offset is in /proc/kallsyms, but not phys_offset. It is just not clear where the
values should come from. I did try using the value of kimage_offset found in
/proc/kallsyms but that did not help. It is not clear where one needs to get the values
from (and in fact a casual reading of the crash source seems to try to compute the values
automatically)
Here is some additional information:
# uname -a
Linux bringup 5.1.0 #1 SMP Fri Oct 11 17:01:55 UTC 2019 aarch64 GNU/Linux
# crash -s
crash: /proc/kcore: No such file or directory
Seem like your kernel is not configured with CONFIG_PROC_KCORE
(/proc/kcore support).
With /proc/kcore you don't need to supply any additional options &
values to crash.
WARNING: could not find MAGIC_START!
WARNING: cannot read linux_banner string
crash: /boot/vmlinux-5.1.0 and /dev/mem do not match!
This is with additional debugging turned on:
# ./crash --version
crash 7.2.8++
Copyright (C) 2002-2020 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu".
Here is crash running with -d flag:
find_booted_kernel: check: /boot/vmlinux-5.1.0
find_booted_kernel: found: /boot/vmlinux-5.1.0
get_live_memory_source: /dev/mem
/proc/version:
Linux version 5.1.0 (oe-user@oe-host) (gcc version 7.3.0 (GCC)) #1 SMP Fri Oct 11
17:01:55 UTC 2019
/boot/vmlinux-5.1.0:
Linux version 5.1.0 (oe-user@oe-host) (gcc version 7.3.0 (GCC)) #1 SMP Fri Oct 11
17:01:55 UTC 2019
crash: /proc/kcore: No such file or directory
readmem: read_dev_mem() -> /dev/mem
VA_BITS: 48
kimage_voffset: fffeffff88000000
phys_offset: 0
physvirt_offset: 800000000000
gdb --quiet /boot/vmlinux-5.1.0
GETBUF(328 -> 0)
GETBUF(1500 -> 1)
FREEBUF(1)
FREEBUF(0)
<readmem: ffff0000109c63b8, KVADDR, "kernel_config_data", 32768, (ROE),
aaaad82d4630>
<read_dev_mem: addr: ffff0000109c63b8 paddr: 889c63b8 cnt: 3144>
<read_dev_mem: addr: ffff0000109c7000 paddr: 889c7000 cnt: 4096>
<read_dev_mem: addr: ffff0000109c8000 paddr: 889c8000 cnt: 4096>
<read_dev_mem: addr: ffff0000109c9000 paddr: 889c9000 cnt: 4096>
<read_dev_mem: addr: ffff0000109ca000 paddr: 889ca000 cnt: 4096>
<read_dev_mem: addr: ffff0000109cb000 paddr: 889cb000 cnt: 4096>
<read_dev_mem: addr: ffff0000109cc000 paddr: 889cc000 cnt: 4096>
<read_dev_mem: addr: ffff0000109cd000 paddr: 889cd000 cnt: 4096>
<read_dev_mem: addr: ffff0000109ce000 paddr: 889ce000 cnt: 952>
<readmem: ffff0000109c63b0, KVADDR, "kernel_config_data MAGIC_START", 8,
(ROE), ffffe6a6d5a0>
<read_dev_mem: addr: ffff0000109c63b0 paddr: 889c63b0 cnt: 8>
WARNING: could not find MAGIC_START!
GETBUF(328 -> 0)
FREEBUF(0)
GETBUF(8 -> 0)
<readmem: ffff000010e1c6d8, KVADDR, "possible", 8, (ROE|Q),
aaaacd15f258>
<read_dev_mem: addr: ffff000010e1c6d8 paddr: 88e1c6d8 cnt: 8>
cpu_possible_mask: cpus: (none)
<readmem: ffff000010e1c6d0, KVADDR, "present", 8, (ROE|Q), aaaacd15f258>
<read_dev_mem: addr: ffff000010e1c6d0 paddr: 88e1c6d0 cnt: 8>
cpu_present_mask: cpus: (none)
<readmem: ffff000010e1c6c8, KVADDR, "online", 8, (ROE|Q), aaaacd15f258>
<read_dev_mem: addr: ffff000010e1c6c8 paddr: 88e1c6c8 cnt: 8>
cpu_online_mask: cpus: (none)
<readmem: ffff000010e1c6e0, KVADDR, "active", 8, (ROE|Q), aaaacd15f258>
<read_dev_mem: addr: ffff000010e1c6e0 paddr: 88e1c6e0 cnt: 8>
cpu_active_mask: cpus: (none)
FREEBUF(0)
GETBUF(328 -> 0)
FREEBUF(0)
GETBUF(328 -> 0)
FREEBUF(0)
GETBUF(328 -> 0)
FREEBUF(0)
<readmem: ffff000010ef8150, KVADDR, "shadow_timekeeper xtime_sec", 8, (ROE),
ffffe6a650b8>
<read_dev_mem: addr: ffff000010ef8150 paddr: 88ef8150 cnt: 8>
xtime timespec.tv_sec: 0: Wed Dec 31 16:00:00 1969
<readmem: ffff000010e2533c, KVADDR, "init_uts_ns", 390, (ROE),
aaaacd0fd194>
<read_dev_mem: addr: ffff000010e2533c paddr: 88e2533c cnt: 390>
utsname:
sysname:
nodename:
release:
version:
machine:
domainname:
base kernel version: 0.0.0
<readmem: ffff0000109b0098, KVADDR, "accessible check", 8, (ROE|Q),
ffffe6a64770>
<read_dev_mem: addr: ffff0000109b0098 paddr: 889b0098 cnt: 8>
<readmem: ffff0000109b0098, KVADDR, "read_string characters", 1499, (ROE|Q),
ffffe6a64ae8>
<read_dev_mem: addr: ffff0000109b0098 paddr: 889b0098 cnt: 1499>
WARNING: cannot read linux_banner string
/proc/version:
Linux version 5.1.0 (oe-user@oe-host) (gcc version 7.3.0 (GCC)) #1 SMP Fri Oct 11
17:01:55 UTC 2019
linux_banner:
crash: /boot/vmlinux-5.1.0 and /dev/mem do not match!
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
8:09 a.m.
New subject: Crash issues with ARM aarch64 on a live system
----- Original Message -----
On Wed, May 20, 2020 at 10:19 PM Siamak Nazari
<siamaknazari(a)yahoo.com>
wrote:
>
> Apologies if this the wrong place to ask this question. If in fact this is
> the wrong alias to post this question, appreciate a pointer in the right
> direction. We have been trying to get crash working on a live kernel
> (aarch64) but have not been able to.
>
> Based on the crash help text:
>
> --machdep option=value
> Pass an option and value pair to machine-dependent code. These
> architecture-specific option/pairs should only be required in
> very rare circumstances:
>
> ARM64:
> phys_offset=<physical-address>
> kimage_voffset=<kimage_voffset-value>
> max_physmem_bits=<value>
> vabits_actual=<value>
>
> We probably need to specify the proper values for phys_offset and
> kimage_voffset. kimage_offset is in /proc/kallsyms, but not phys_offset.
> It is just not clear where the values should come from. I did try using
> the value of kimage_offset found in /proc/kallsyms but that did not help.
> It is not clear where one needs to get the values from (and in fact a
> casual reading of the crash source seems to try to compute the values
> automatically)
>
> Here is some additional information:
>
> # uname -a
> Linux bringup 5.1.0 #1 SMP Fri Oct 11 17:01:55 UTC 2019 aarch64 GNU/Linux
>
> # crash -s
> crash: /proc/kcore: No such file or directory
Seem like your kernel is not configured with CONFIG_PROC_KCORE (/proc/kcore support).
With /proc/kcore you don't need to supply any additional options &
values to crash.
Santosh is correct, with /proc/kcore things should just work.
On your system, it then tries /dev/mem, which needs help, including phys_offset.
You might try getting it from /proc/iomem, where on my arm64 system, phys_offset
is 0x4000000000:
$ cat /proc/iomem
...
7e890000-7e890fff : APMC0D60:02
7e8d0000-7e8d0fff : APMC0D60:03
7e940000-7e940fff : APMC0D5E:00
4000000000-40001fffff : reserved
4000200000-43fa59ffff : System RAM
4000280000-40012effff : Kernel code
40012f0000-40017fffff : reserved
4001800000-4002ccffff : Kernel data
40dfe00000-40e3dfffff : reserved
40e3e00000-40ffdfffff : Crash kernel
40ffff0000-40ffffffff : reserved
41909d0000-41919cffff : reserved
43909d0000-4393ffffff : reserved
43f8750000-43f875ffff : reserved
...
Try adding that on the command line.
Dave
> WARNING: could not find MAGIC_START!
> WARNING: cannot read linux_banner string
> crash: /boot/vmlinux-5.1.0 and /dev/mem do not match!
>
> This is with additional debugging turned on:
>
> # ./crash --version
>
> crash 7.2.8++
> Copyright (C) 2002-2020 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
> and "show warranty" for details.
> This GDB was configured as "aarch64-unknown-linux-gnu".
>
> Here is crash running with -d flag:
>
> find_booted_kernel: check: /boot/vmlinux-5.1.0
> find_booted_kernel: found: /boot/vmlinux-5.1.0
> get_live_memory_source: /dev/mem
> /proc/version:
> Linux version 5.1.0 (oe-user@oe-host) (gcc version 7.3.0 (GCC)) #1 SMP Fri
> Oct 11 17:01:55 UTC 2019
> /boot/vmlinux-5.1.0:
> Linux version 5.1.0 (oe-user@oe-host) (gcc version 7.3.0 (GCC)) #1 SMP Fri
> Oct 11 17:01:55 UTC 2019
> crash: /proc/kcore: No such file or directory
> readmem: read_dev_mem() -> /dev/mem
> VA_BITS: 48
> kimage_voffset: fffeffff88000000
> phys_offset: 0
> physvirt_offset: 800000000000
> gdb --quiet /boot/vmlinux-5.1.0
> GETBUF(328 -> 0)fffeffc00fe00000
> GETBUF(1500 -> 1)
> FREEBUF(1)
> FREEBUF(0)
> <readmem: ffff0000109c63b8, KVADDR, "kernel_config_data", 32768,
(ROE),
> aaaad82d4630>
> <read_dev_mem: addr: ffff0000109c63b8 paddr: 889c63b8 cnt: 3144>
> <read_dev_mem: addr: ffff0000109c7000 paddr: 889c7000 cnt: 4096>
> <read_dev_mem: addr: ffff0000109c8000 paddr: 889c8000 cnt: 4096>
> <read_dev_mem: addr: ffff0000109c9000 paddr: 889c9000 cnt: 4096>
> <read_dev_mem: addr: ffff0000109ca000 paddr: 889ca000 cnt: 4096>
> <read_dev_mem: addr: ffff0000109cb000 paddr: 889cb000 cnt: 4096>
> <read_dev_mem: addr: ffff0000109cc000 paddr: 889cc000 cnt: 4096>
> <read_dev_mem: addr: ffff0000109cd000 paddr: 889cd000 cnt: 4096>
> <read_dev_mem: addr: ffff0000109ce000 paddr: 889ce000 cnt: 952>
> <readmem: ffff0000109c63b0, KVADDR, "kernel_config_data MAGIC_START",
8,
> (ROE), ffffe6a6d5a0>
> <read_dev_mem: addr: ffff0000109c63b0 paddr: 889c63b0 cnt: 8>
> WARNING: could not find MAGIC_START!
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(8 -> 0)
> <readmem: ffff000010e1c6d8, KVADDR, "possible", 8, (ROE|Q),
aaaacd15f258>
> <read_dev_mem: addr: ffff000010e1c6d8 paddr: 88e1c6d8 cnt: 8>
> cpu_possible_mask: cpus: (none)
> <readmem: ffff000010e1c6d0, KVADDR, "present", 8, (ROE|Q),
aaaacd15f258>
> <read_dev_mem: addr: ffff000010e1c6d0 paddr: 88e1c6d0 cnt: 8>
> cpu_present_mask: cpus: (none)
> <readmem: ffff000010e1c6c8, KVADDR, "online", 8, (ROE|Q),
aaaacd15f258>
> <read_dev_mem: addr: ffff000010e1c6c8 paddr: 88e1c6c8 cnt: 8>
> cpu_online_mask: cpus: (none)
> <readmem: ffff000010e1c6e0, KVADDR, "active", 8, (ROE|Q),
aaaacd15f258>
> <read_dev_mem: addr: ffff000010e1c6e0 paddr: 88e1c6e0 cnt: 8>
> cpu_active_mask: cpus: (none)
> FREEBUF(0)
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(328 -> 0)
> FREEBUF(0)
> <readmem: ffff000010ef8150, KVADDR, "shadow_timekeeper xtime_sec", 8,
> (ROE), ffffe6a650b8>
> <read_dev_mem: addr: ffff000010ef8150 paddr: 88ef8150 cnt: 8>
> xtime timespec.tv_sec: 0: Wed Dec 31 16:00:00 1969
> <readmem: ffff000010e2533c, KVADDR, "init_uts_ns", 390, (ROE),
> aaaacd0fd194>
> <read_dev_mem: addr: ffff000010e2533c paddr: 88e2533c cnt: 390>
> utsname:
> sysname:
> nodename:
> release:
> version:
> machine:
> domainname:
> base kernel version: 0.0.0
> <readmem: ffff0000109b0098, KVADDR, "accessible check", 8, (ROE|Q),
> ffffe6a64770>
> <read_dev_mem: addr: ffff0000109b0098 paddr: 889b0098 cnt: 8>
> <readmem: ffff0000109b0098, KVADDR, "read_string characters", 1499,
> (ROE|Q), ffffe6a64ae8>
> <read_dev_mem: addr: ffff0000109b0098 paddr: 889b0098 cnt: 1499>
> WARNING: cannot read linux_banner string
> /proc/version:
> Linux version 5.1.0 (oe-user@oe-host) (gcc version 7.3.0 (GCC)) #1 SMP Fri
> Oct 11 17:01:55 UTC 2019
> linux_banner:
>
> crash: /boot/vmlinux-5.1.0 and /dev/mem do not match!
>
>
>
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
1502
days inactive
1530
days old
devel@lists.crash-utility.osci.io
8 comments
5 participants
participants (5)
-
Dave Anderson -
HAGIO KAZUHITO(萩尾 一仁) -
John Ogness -
Santosh -
Siamak Nazari