Re: [Crash-utility] [Patch-v2] Fix handling the ring buffer for 'log' command.
by Dave Anderson
----- "Ken'ichi Ohmichi" <oomichi(a)mxs.nes.nec.co.jp> wrote:
> Hi Dave, Jeff,
>
> Thank you for comments.
>
> Jeff Moyer wrote:
> > > Dave Anderson <anderson(a)redhat.com> writes:
> > >
> >> >> Good question -- I'm not sure.
> >> >>
> >> >> If the buffer were "cleared" by the administrator, the logical "end" of
> >> >> the buffer would not be the last thing displayed by the "log" command.
> >> >> But that's really not a problem, given that the relevant kernel-crash-related
> >> >> data is still available to be examined.
> >> >>
> >> >> On the other hand, even though the administrator has "cleared" the log
> >> >> buffer, the data is still there. My concern would be what if there is
> >> >> crucial data in the log buffer that the administrator "cleared" out of
> >> >> convenience? Like for example, I've often done a "dmesg -c" to clear
> >> >> the buffer so that subsequent dmesg commands just dump the latest
> >> >> information. But then I've gone back with the crash utility and
> >> >> re-examined the log buffer data that still remains in memory -- which
> >> >> can be still be seen with the "log" command.
>
> Good comment, I see.
>
>
> >> >> So my initial leaning would be to continue to show what's actually there.
> >> >> I trust myself as a crash analyzer more than I trust the administrator.
> >> >>
> >> >> But I could be convinced otherwise.
> >> >>
> >> >> What do others on the list think about this?
> > >
> > > I would much rather see all of the contents of the log buffer.
>
> OK, I attach a new patch for outputing all of the contents of the log
> buffer. How about this patch ?
It appears to works OK, although the leftover references to "logged_chars"
should be removed from your patch, correct?
Dave
>
>
> By the way, I'm implementing a new option '--dump-dmesg' of makedumpfile
> command with Neil Horman. The option is similar to 'log' command of the
> crash utility, so the option extracts log data from /proc/vmcore and dumps
> it to a file. I will implement the option for outputting all of the contents
> of the log buffer like this patch.
>
>
> ---
> [Patch-v2] Fix handling the ring buffer for 'log' command.
>
> Changelog of v2:
> - Extending the output range of 'log' command to all the ring buffer.
>
> I found there is a 'log' command problem related to handling the ring
> buffer, and this patch fixes it.
>
> The ring buffer can be cleared by klogctl(2) from a user process, but
> current crash utility does not consider this case. The following output
> of current crash utility is example of this case. You see there are some
> messages after the panic message. These messages are the oldest log data,
> so they should be output first.
>
> crash > log
> [snip]
> exception[8291] trap divide error rip:4004c0 rsp:7fffb56cdf60 error:0
> exception[8293] trap divide error rip:4004c0 rsp:7fffbb0628f0 error:0
> SysRq : Trigger a crashdump
> 14 15) *0, disabled.
> ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 6 7 10 14 15) *0, disabled.
> ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 14 15) *0, disabled.
> [snip]
> crash >
>
> This patch fixes this invalid output to the folloing:
>
> crash > log
> 14 15) *0, disabled.
> ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 6 7 10 14 15) *0, disabled.
> ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 14 15) *0, disabled.
> [snip]
> exception[8291] trap divide error rip:4004c0 rsp:7fffb56cdf60 error:0
> exception[8293] trap divide error rip:4004c0 rsp:7fffbb0628f0 error:0
> SysRq : Trigger a crashdump
> crash>
>
>
> Thanks
> Ken'ichi Ohmichi
>
> Signed-off-by: Ken'ichi Ohmichi <oomichi(a)mxs.nes.nec.co.jp>
> ---
> --- crash-4.0-7.7/kernel.c.orig 2009-02-06 01:13:43.000000000 +0900
> +++ crash-4.0-7.7/kernel.c 2009-02-18 10:42:29.000000000 +0900
> @@ -3365,7 +3365,7 @@ void
> dump_log(int msg_level)
> {
> int i;
> - ulong log_buf, logged_chars;
> + ulong log_buf, logged_chars, log_end;
> char *buf;
> char last;
> ulong index;
> @@ -3393,17 +3393,16 @@ dump_log(int msg_level)
> buf = GETBUF(log_buf_len);
> log_wrap = FALSE;
> get_symbol_data("logged_chars", sizeof(ulong), &logged_chars);
> + get_symbol_data("log_end", sizeof(ulong), &log_end);
> readmem(log_buf, KVADDR, buf,
> log_buf_len, "log_buf contents", FAULT_ON_ERROR);
>
> - if (logged_chars < log_buf_len) {
> + if (log_end < log_buf_len)
> index = 0;
> - } else {
> - get_symbol_data("log_end", sizeof(ulong), &index);
> - index &= log_buf_len-1;
> - }
> + else
> + index = log_end & (log_buf_len - 1);
>
> - if ((logged_chars < log_buf_len) && (index == 0) && (buf[index] ==
> '<'))
> + if ((log_end < log_buf_len) && (index == 0) && (buf[index] == '<'))
> loglevel = TRUE;
> else
> loglevel = FALSE;
>
17 years, 1 month
Re: [Crash-utility] [Patch] Fix handling the ring buffer for 'log' command.
by Dave Anderson
----- "Ken'ichi Ohmichi" <oomichi(a)mxs.nes.nec.co.jp> wrote:
> Hi,
>
> I found there is a 'log' command problem related to handling the ring
> buffer, and this patch fixes it.
>
> The ring buffer can be cleared by klogctl(2) from a user process, but
> current crash utility does not consider this case. The following output
> of current crash utility is example of this case. You see there is the
> panic message "SysRq : Trigger a crashdump" inside even if it should be
> output in the end:
>
> crash > log
> [snip]
> exception[8291] trap divide error rip:4004c0 rsp:7fffb56cdf60 error:0
> exception[8293] trap divide error rip:4004c0 rsp:7fffbb0628f0 error:0
> SysRq : Trigger a crashdump
> 14 15) *0, disabled.
> ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 6 7 10 14 15) *0, disabled.
> ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 14 15) *0, disabled.
> [snip]
> crash>
>
> This patch fixes this invalid output to the folloing:
>
> crash> log
> [snip]
> exception[8291] trap divide error rip:4004c0 rsp:7fffb56cdf60 error:0
> exception[8293] trap divide error rip:4004c0 rsp:7fffbb0628f0 error:0
> SysRq : Trigger a crashdump
> crash>
>
> I have one concern about this fixing:
> If applying this patch, 'log' command outputs the log data from the time
> of klogctl(2) call for the log clearance. If making the other patch, we
> will be able to extract the former log data, because a kernel does not
> clear the ring buffer and it just clears "logged_chars" internally.
> I feel this patch is better, because the log data can be cleared only by
> a root user and he decided the former log data is unnecessary.
> What do you think of this?
>
Good question -- I'm not sure.
If the buffer were "cleared" by the administrator, the logical "end" of
the buffer would not be the last thing displayed by the "log" command.
But that's really not a problem, given that the relevant kernel-crash-related
data is still available to be examined.
On the other hand, even though the administrator has "cleared" the log
buffer, the data is still there. My concern would be what if there is
crucial data in the log buffer that the administrator "cleared" out of
convenience? Like for example, I've often done a "dmesg -c" to clear
the buffer so that subsequent dmesg commands just dump the latest
information. But then I've gone back with the crash utility and
re-examined the log buffer data that still remains in memory -- which
can be still be seen with the "log" command.
So my initial leaning would be to continue to show what's actually there.
I trust myself as a crash analyzer more than I trust the administrator.
But I could be convinced otherwise.
What do others on the list think about this?
Dave
>
> Thanks
> Ken'ichi Ohmichi
>
> Signed-off-by: Ken'ichi Ohmichi <oomichi(a)mxs.nes.nec.co.jp>
> ---
> --- crash-4.0-7.7/kernel.c.orig 2009-02-06 01:13:43.000000000 +0900
> +++ crash-4.0-7.7/kernel.c 2009-02-17 12:07:33.000000000 +0900
> @@ -3364,13 +3364,13 @@ cmd_log(void)
> void
> dump_log(int msg_level)
> {
> - int i;
> + int i, j;
> ulong log_buf, logged_chars;
> char *buf;
> char last;
> ulong index;
> struct syment *nsp;
> - int log_wrap, loglevel, log_buf_len;
> + int loglevel, log_buf_len;
>
> if (symbol_exists("log_buf_len")) {
> get_symbol_data("log_buf_len", sizeof(int), &log_buf_len);
> @@ -3391,29 +3391,28 @@ dump_log(int msg_level)
> }
>
> buf = GETBUF(log_buf_len);
> - log_wrap = FALSE;
> get_symbol_data("logged_chars", sizeof(ulong), &logged_chars);
> readmem(log_buf, KVADDR, buf,
> log_buf_len, "log_buf contents", FAULT_ON_ERROR);
>
> + get_symbol_data("log_end", sizeof(ulong), &index);
> + index &= (log_buf_len - 1);
> if (logged_chars < log_buf_len) {
> - index = 0;
> - } else {
> - get_symbol_data("log_end", sizeof(ulong), &index);
> - index &= log_buf_len-1;
> - }
> + if (logged_chars <= index)
> + index -= logged_chars;
> + else
> + index = log_buf_len - (logged_chars - index);
> + }
>
> if ((logged_chars < log_buf_len) && (index == 0) && (buf[index] == '<'))
> loglevel = TRUE;
> else
> loglevel = FALSE;
>
> - if (index != 0)
> - log_wrap = TRUE;
> + for (i = index, j = 0; j < logged_chars; i++, j++) {
> + if (i >= log_buf_len)
> + i = 0;
>
> -wrap_around:
> -
> - for (i = index; i < log_buf_len; i++) {
> if (loglevel && !msg_level) {
> switch (buf[i])
> {
> @@ -3444,13 +3443,6 @@ wrap_around:
> }
> }
>
> - if (log_wrap) {
> - log_buf_len = index;
> - index = 0;
> - log_wrap = FALSE;
> - goto wrap_around;
> - }
> -
> if (last != '\n')
> fprintf(fp, "\n");
>
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
17 years, 1 month
[Patch] Fix handling the ring buffer for 'log' command.
by Ken'ichi Ohmichi
Hi,
I found there is a 'log' command problem related to handling the ring
buffer, and this patch fixes it.
The ring buffer can be cleared by klogctl(2) from a user process, but
current crash utility does not consider this case. The following output
of current crash utility is example of this case. You see there is the
panic message "SysRq : Trigger a crashdump" inside even if it should be
output in the end:
crash > log
[snip]
exception[8291] trap divide error rip:4004c0 rsp:7fffb56cdf60 error:0
exception[8293] trap divide error rip:4004c0 rsp:7fffbb0628f0 error:0
SysRq : Trigger a crashdump
14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 6 7 10 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 14 15) *0, disabled.
[snip]
crash >
This patch fixes this invalid output to the folloing:
crash > log
[snip]
exception[8291] trap divide error rip:4004c0 rsp:7fffb56cdf60 error:0
exception[8293] trap divide error rip:4004c0 rsp:7fffbb0628f0 error:0
SysRq : Trigger a crashdump
crash>
I have one concern about this fixing:
If applying this patch, 'log' command outputs the log data from the time
of klogctl(2) call for the log clearance. If making the other patch, we
will be able to extract the former log data, because a kernel does not
clear the ring buffer and it just clears "logged_chars" internally.
I feel this patch is better, because the log data can be cleared only by
a root user and he decided the former log data is unnecessary.
What do you think of this?
Thanks
Ken'ichi Ohmichi
Signed-off-by: Ken'ichi Ohmichi <oomichi(a)mxs.nes.nec.co.jp>
---
--- crash-4.0-7.7/kernel.c.orig 2009-02-06 01:13:43.000000000 +0900
+++ crash-4.0-7.7/kernel.c 2009-02-17 12:07:33.000000000 +0900
@@ -3364,13 +3364,13 @@ cmd_log(void)
void
dump_log(int msg_level)
{
- int i;
+ int i, j;
ulong log_buf, logged_chars;
char *buf;
char last;
ulong index;
struct syment *nsp;
- int log_wrap, loglevel, log_buf_len;
+ int loglevel, log_buf_len;
if (symbol_exists("log_buf_len")) {
get_symbol_data("log_buf_len", sizeof(int), &log_buf_len);
@@ -3391,29 +3391,28 @@ dump_log(int msg_level)
}
buf = GETBUF(log_buf_len);
- log_wrap = FALSE;
get_symbol_data("logged_chars", sizeof(ulong), &logged_chars);
readmem(log_buf, KVADDR, buf,
log_buf_len, "log_buf contents", FAULT_ON_ERROR);
+ get_symbol_data("log_end", sizeof(ulong), &index);
+ index &= (log_buf_len - 1);
if (logged_chars < log_buf_len) {
- index = 0;
- } else {
- get_symbol_data("log_end", sizeof(ulong), &index);
- index &= log_buf_len-1;
- }
+ if (logged_chars <= index)
+ index -= logged_chars;
+ else
+ index = log_buf_len - (logged_chars - index);
+ }
if ((logged_chars < log_buf_len) && (index == 0) && (buf[index] == '<'))
loglevel = TRUE;
else
loglevel = FALSE;
- if (index != 0)
- log_wrap = TRUE;
+ for (i = index, j = 0; j < logged_chars; i++, j++) {
+ if (i >= log_buf_len)
+ i = 0;
-wrap_around:
-
- for (i = index; i < log_buf_len; i++) {
if (loglevel && !msg_level) {
switch (buf[i])
{
@@ -3444,13 +3443,6 @@ wrap_around:
}
}
- if (log_wrap) {
- log_buf_len = index;
- index = 0;
- log_wrap = FALSE;
- goto wrap_around;
- }
-
if (last != '\n')
fprintf(fp, "\n");
17 years, 1 month
Remote crash?
by Jun Koi
Hi,
It seems that remote analysing crash is not supported any more? What
is the reason behind this decision?
If this is true, what is the final version with remote support work well?
Thanks,
Jun
17 years, 1 month
Re: [Crash-utility] crash-extensions rpm
by Dave Anderson
----- "Michael Holzheu" <holzheu(a)linux.vnet.ibm.com> wrote:
> Hi Dave,
>
> What about including the extensions (especially sial) in the crash rpm?
> Or maybe even better, create a second crash-extensions sub package.
>
> Michael
>
> Here a proposal for the SPEC file change:
Looks reasonable to me.
Although this change will only be reflected in the upstream src.rpm file
on my people site, there is sort of a precedent in RHEL5. There is a RHEL5
ppc64 Cell/B.E. SPU "crash-spu-commands" package, which is a standalone crash
extension package, and which puts its .so file in /usr/lib64/crash/extensions:
$ rpm -qpl crash-spu-commands-1.1-1.ppc64.rpm
/usr/lib64/crash/extensions/spu.so
/usr/share/doc/crash-spu-commands-1.1
/usr/share/doc/crash-spu-commands-1.1/COPYING
$
So its .spec file has this:
%install
mkdir -p %{buildroot}%{_libdir}/crash/extensions/
cp %{_builddir}/%{name}-%{version}/spu.so %{buildroot}%{_libdir}/crash/extensions/
and this:
%files
%defattr(-,root,root)
%{_libdir}/crash/extensions/spu.so
%doc COPYING
That should work for you right, i.e., changing your "/usr/lib" references
to %{_libdir}?
Dave
> ---
> --- crash.spec.orig 2009-02-12 13:05:11.000000000 +0100
> +++ crash.spec 2009-02-12 14:38:11.000000000 +0100
> @@ -36,12 +36,25 @@
> netdump, diskdump and kdump packages from Red Hat Linux, the mcore kernel patch
> offered by Mission Critical Linux, or the LKCD kernel patch.
>
> +%package extensions
> +Summary: Additional commands for the crash dump analysis tool
> +Group: Development/Debuggers
> +
> +%description extensions
> +The extensions package contains plugins that provide additional crash
> +commands. The extensions can be loaded in crash via the "extend" command.
> +
> +The following extensions are provided:
> +* sial: Provides C-like language for writing dump analysis scripts
> +* dminfo: Device-mapper target analyzer
> +
> %prep
> %setup -n %{name}-%{version}-%{release}
> # %patch0 -p1 -b .install (patch example)
>
> %build
> make RPMPKG="%{version}-%{release}"
> +make RPMPKG="%{version}-%{release}" extensions
>
> %install
> rm -rf %{buildroot}
> @@ -51,6 +64,8 @@
> cp crash.8 %{buildroot}%{_mandir}/man8/crash.8
> mkdir -p %{buildroot}%{_includedir}/crash
> cp defs.h %{buildroot}%{_includedir}/crash
> +mkdir -p %{buildroot}/usr/lib/crash
> +cp extensions/*.so %{buildroot}/usr/lib/crash
>
> %clean
> rm -rf %{buildroot}
> @@ -63,3 +78,7 @@
> %files devel
> %defattr(-,root,root)
> %{_includedir}/*
> +
> +%files extensions
> +%defattr(-,root,root)
> +/usr/lib/crash/*
>
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
17 years, 1 month
crash-extensions rpm
by Michael Holzheu
Hi Dave,
What about including the extensions (especially sial) in the crash rpm?
Or maybe even better, create a second crash-extensions sub package.
Michael
Here a proposal for the SPEC file change:
---
--- crash.spec.orig 2009-02-12 13:05:11.000000000 +0100
+++ crash.spec 2009-02-12 14:38:11.000000000 +0100
@@ -36,12 +36,25 @@
netdump, diskdump and kdump packages from Red Hat Linux, the mcore kernel patch
offered by Mission Critical Linux, or the LKCD kernel patch.
+%package extensions
+Summary: Additional commands for the crash dump analysis tool
+Group: Development/Debuggers
+
+%description extensions
+The extensions package contains plugins that provide additional crash
+commands. The extensions can be loaded in crash via the "extend" command.
+
+The following extensions are provided:
+* sial: Provides C-like language for writing dump analysis scripts
+* dminfo: Device-mapper target analyzer
+
%prep
%setup -n %{name}-%{version}-%{release}
# %patch0 -p1 -b .install (patch example)
%build
make RPMPKG="%{version}-%{release}"
+make RPMPKG="%{version}-%{release}" extensions
%install
rm -rf %{buildroot}
@@ -51,6 +64,8 @@
cp crash.8 %{buildroot}%{_mandir}/man8/crash.8
mkdir -p %{buildroot}%{_includedir}/crash
cp defs.h %{buildroot}%{_includedir}/crash
+mkdir -p %{buildroot}/usr/lib/crash
+cp extensions/*.so %{buildroot}/usr/lib/crash
%clean
rm -rf %{buildroot}
@@ -63,3 +78,7 @@
%files devel
%defattr(-,root,root)
%{_includedir}/*
+
+%files extensions
+%defattr(-,root,root)
+/usr/lib/crash/*
17 years, 1 month
[PATCH] remove abundant code in memory.c
by Jun Koi
Hi,
This patch removes abundant code in memory.c
Thanks,
Jun
diff a/memory.c b/memory.c
--- a/memory.c
+++ b/memory.c
@@ -3369,7 +3369,6 @@ clear_vma_cache(void)
void
get_task_mem_usage(ulong task, struct task_mem_usage *tm)
{
- int rdflags;
struct task_context *tc;
BZERO(tm, sizeof(struct task_mem_usage));
@@ -3377,8 +3376,6 @@ get_task_mem_usage(ulong task, struct task_mem_usage *tm)
if (IS_ZOMBIE(task) || IS_EXITING(task))
return;
- rdflags = ACTIVE() ? (QUIET|RETURN_ON_ERROR) : RETURN_ON_ERROR;
-
tc = task_to_context(task);
if (!tc || !tc->mm_struct) /* probably a kernel thread */
17 years, 1 month
Re: [Crash-utility] crash bug on 2.6.27.8
by Dave Anderson
----- "Jun Koi" <junkoi2004(a)gmail.com> wrote:
> Hi,
>
> When using the latest crash on 2.6.27.14 to analyze live memory
> (running crash without any option), I got a bug like below. How can I
> fix that?
>
> (A similar problem is also seen on 2.6.27.8)
>
> Thanks,
> Jun
The "read error" indications are most likely due to your kernel
being configured with CONFIG_STRICT_DEVMEM, which restricts
/dev/mem access to the first 256 pages of physical memory,
which makes it useless for the crash utility.
If you cannot reconfigure your kernel without it, then you
have two other options.
1. Port the RHEL/Fedora /dev/crash (drivers/char/crash.c) memory
driver, and insmod it prior to invoking crash. Red Hat kernels
have had the /dev/mem restriction in place by default since
RHEL4. It's only fairly recently that it's been imposed by
default in upstream kernels.
2. You can also write a simple kprobe module module that
modifies the kernel's devmem_is_allowed() function to
always return 1. That will fake the /dev/mem driver into
accepting all physical address requests.
I don't have the links off-hand, but if you search through
this mailing list's archives, say since the September 2008
timeframe, you can find threads that that describe how (1)
and (2) can be accomplished.
But if it all possible, the easiest way to go is by reconfiguring
your kernel with CONFIG_STRICT_DEVMEM turned off.
I'm going to be out of touch for a couple of days, so I won't be
of much further help until Wednesday. If anybody else on the list
has done this, please give this guy any pointers you might have.
Thanks,
Dave
>
> .....
> GNU gdb 6.1
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and
> you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for
> details.
> This GDB was configured as "i686-pc-linux-gnu"...
>
> crash: read error: kernel virtual address: c0488200 type:
> "cpu_possible_map"
> WARNING: cannot read cpu_possible_map
> crash: read error: kernel virtual address: c041b428 type:
> "cpu_present_map"
> WARNING: cannot read cpu_present_map
> crash: read error: kernel virtual address: c041ab00 type:
> "cpu_online_map"
> WARNING: cannot read cpu_online_map
> crash: read error: kernel virtual address: c04b1900 type: "xtime"
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
17 years, 1 month
minimal mode?
by Jun Koi
Hi,
crash can run in minimal mode, which provides only limited commands. I
am wondering what is the motivation behind this mode?
Thanks,
Jun
17 years, 1 month
