5:04 a.m.
Hi Experts:
Sorry for disturbing you again.
When I run crash to debug vmcore, if kernel enable kgdb, it will get into kdb shell
directly not crash shell, althrough I can get into crash via execute two go command in kdb
shell, not sure if it's normal, do you have some comments on it?
When I disnable kgdb in kernel, there will not be this kind of issue, if there is some
compatibility problem between kgdb and crash tool?
Thanks for your time and happen new year!!
/****************************************/
root@localhost:~/coredump>sh testcoredump.sh; sh startcoredump.sh
segment[0].mem:0x2000000 memsz:9183232
segment[1].mem:0x28c2000 memsz:65536
segment[2].mem:0x28d2000 memsz:4096
segment[3].mem:0x28d3000 memsz:28672
segment[4].mem:0x2ff5000 memsz:45056
SysRq : Trigger a crash
Entering kdb (current=0xdb17f900, pid 822) on processor 1 Oops: (null)
due to oops @ 0xc0348f64
NIP: c0348f64 LR: c034974c CTR: c0348f50
REGS: db18ddc0 TRAP: 0300 Not tainted (2.6.34.15-grsec-WR4.3.0.0_cgl)
MSR: 00021202 <ME,CE,DE> CR: 20242444 XER: 00000000
DEAR: 00000000, ESR: 00800000
TASK = db17f900[822] 'sh' THREAD: db18c000 CPU: 1
GPR00: 00000001 db18de70 db17f900 00000063 00000000 ffffffff c035e930 00000000
GPR08: 00008000 00000000 00000054 118c6000 20242444 100b84b8 100b0828 100b0904
GPR16: 00000000 00000000 00000000 1006d2d0 00000000 100cc220 1006b530 00000000
GPR24: 00000001 c0780d24 00029202 c0780e08 c0770000 00000000 c08578a4 00000063
NIP [c0348f64] sysrq_handle_crash+0x14/0x20
LR [c034974c] __handle_sysrq+0xcc/0x1d0
Call Trace:
[db18de70] [c0349734] __handle_sysrq+0xb4/0x1d0 (unreliable)
[db18dea0] [c03498ac] write_sysrq_trigger+0x5c/0x70
[db18deb0] [c01662f0] proc_reg_write+0x80/0xc0
[db18dee0] [c010ec50] vfs_write+0xc0/0x170
[db18df00] [c010ee80] sys_write+0x50/0x110
[db18df40] [c0011d58] ret_from_syscall+0x0/0x4
--- Exception: c01 at 0xff16ba4
LR = 0xfebad5c
Instruction dump:
[1]more>
39600003 7d605f9e 556b103a 7c005b78 98090003 4e800020 60000000 38000001
3d20c07a 90092170 7c0004ac 39200000 <98090000> 4e800020 60000000 3803ffd0
[1]kdb>
[1]kdb> go
Catastrophic error detected
kdb_continue_catastrophic=0, type go a second time if you really want to continue
[1]kdb> go
Catastrophic error detected
kdb_continue_catastrophic=0, attempting to continue
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT SMP NR_CPUS=4 LTT NESTING LEVEL : 0
P2041 RDB
last sysfs file: /sys/devices/system/cpu/cpu3/crash_notes
Modules linked in:
NIP: c0348f64 LR: c034974c CTR: c0348f50
REGS: db18ddc0 TRAP: 0300 Not tainted (2.6.34.15-grsec-WR4.3.0.0_cgl)
MSR: 00021202 <ME,CE,DE> CR: 20242444 XER: 00000000
DEAR: 00000000, ESR: 00800000
TASK = db17f900[822] 'sh' THREAD: db18c000 CPU: 1
GPR00: 00000001 db18de70 db17f900 00000063 00000000 ffffffff c035e930 00000000
GPR08: 00008000 00000000 00000054 118c6000 20242444 100b84b8 100b0828 100b0904
GPR16: 00000000 00000000 00000000 1006d2d0 00000000 100cc220 1006b530 00000000
GPR24: 00000001 c0780d24 00029202 c0780e08 c0770000 00000000 c08578a4 00000063
NIP [c0348f64] sysrq_handle_crash+0x14/0x20
LR [c034974c] __handle_sysrq+0xcc/0x1d0
Call Trace:
[db18de70] [c0349734] __handle_sysrq+0xb4/0x1d0 (unreliable)
[db18dea0] [c03498ac] write_sysrq_trigger+0x5c/0x70
[db18deb0] [c01662f0] proc_reg_write+0x80/0xc0
[db18dee0] [c010ec50] vfs_write+0xc0/0x170
[db18df00] [c010ee80] sys_write+0x50/0x110
[db18df40] [c0011d58] ret_from_syscall+0x0/0x4
--- Exception: c01 at 0xff16ba4
LR = 0xfebad5c
Instruction dump:
39600003 7d605f9e 556b103a 7c005b78 98090003 4e800020 60000000 38000001
3d20c07a 90092170 7c0004ac 39200000 <98090000> 4e800020 60000000 3803ffd0
Sending IPI to other cpus...
Bye!
Using P2041 RDB machine description
......
/**********************/
Best Regards,
James
Liu Jianbo | WIND RIVER | Senior Engineer - Technical Support
Tel 86 28 65318098 | Cell 86 13558641588 | Fax 86 28 65319983
Manage your support account:
https://support.windriver.com
Ask a Technical Question
https://ask.windriver.com
Submit a Service Request
https://windriver.force.com/support
8:13 a.m.
On 30 December 2015 at 16:34, Liu, Jianbo (James) <James.Liu(a)windriver.com>
wrote:
Hi Experts:
Sorry for disturbing you again.
When I run crash to debug vmcore, if kernel enable kgdb, it will get into
kdb shell directly not crash shell, althrough I can get into crash via
execute two go command in kdb shell, not sure if it's normal, do you have
some comments on it?
When I disnable kgdb in kernel, there will not be this kind of issue, if
there is some compatibility problem between kgdb and crash tool?
Thanks for your time and happen new year!!
/****************************************/
root@localhost:~/coredump>sh testcoredump.sh; sh startcoredump.sh
segment[0].mem:0x2000000 memsz:9183232
segment[1].mem:0x28c2000 memsz:65536
segment[2].mem:0x28d2000 memsz:4096
segment[3].mem:0x28d3000 memsz:28672
segment[4].mem:0x2ff5000 memsz:45056
SysRq : Trigger a crash
Entering kdb (current=0xdb17f900, pid 822) on processor 1 Oops: (null)
due to oops @ 0xc0348f64
NIP: c0348f64 LR: c034974c CTR: c0348f50
REGS: db18ddc0 TRAP: 0300 Not tainted (2.6.34.15-grsec-WR4.3.0.0_cgl)
MSR: 00021202 <ME,CE,DE> CR: 20242444 XER: 00000000
DEAR: 00000000, ESR: 00800000
TASK = db17f900[822] 'sh' THREAD: db18c000 CPU: 1
GPR00: 00000001 db18de70 db17f900 00000063 00000000 ffffffff c035e930 00000000
GPR08: 00008000 00000000 00000054 118c6000 20242444 100b84b8 100b0828 100b0904
GPR16: 00000000 00000000 00000000 1006d2d0 00000000 100cc220 1006b530 00000000
GPR24: 00000001 c0780d24 00029202 c0780e08 c0770000 00000000 c08578a4 00000063
NIP [c0348f64] sysrq_handle_crash+0x14/0x20
LR [c034974c] __handle_sysrq+0xcc/0x1d0
Call Trace:
[db18de70] [c0349734] __handle_sysrq+0xb4/0x1d0 (unreliable)
[db18dea0] [c03498ac] write_sysrq_trigger+0x5c/0x70
[db18deb0] [c01662f0] proc_reg_write+0x80/0xc0
[db18dee0] [c010ec50] vfs_write+0xc0/0x170
[db18df00] [c010ee80] sys_write+0x50/0x110
[db18df40] [c0011d58] ret_from_syscall+0x0/0x4
--- Exception: c01 at 0xff16ba4
LR = 0xfebad5c
Instruction dump:
[1]more>
39600003 7d605f9e 556b103a 7c005b78 98090003 4e800020 60000000 38000001
3d20c07a 90092170 7c0004ac 39200000 <98090000> 4e800020 60000000 3803ffd0
[1]kdb>
[1]kdb> go
Catastrophic error detected
kdb_continue_catastrophic=0, type go a second time if you really want to continue
[1]kdb> go
Catastrophic error detected
kdb_continue_catastrophic=0, attempting to continue
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT SMP NR_CPUS=4 LTT NESTING LEVEL : 0
P2041 RDB
last sysfs file: /sys/devices/system/cpu/cpu3/crash_notes
Modules linked in:
NIP: c0348f64 LR: c034974c CTR: c0348f50
REGS: db18ddc0 TRAP: 0300 Not tainted (2.6.34.15-grsec-WR4.3.0.0_cgl)
MSR: 00021202 <ME,CE,DE> CR: 20242444 XER: 00000000
DEAR: 00000000, ESR: 00800000
TASK = db17f900[822] 'sh' THREAD: db18c000 CPU: 1
GPR00: 00000001 db18de70 db17f900 00000063 00000000 ffffffff c035e930 00000000
GPR08: 00008000 00000000 00000054 118c6000 20242444 100b84b8 100b0828 100b0904
GPR16: 00000000 00000000 00000000 1006d2d0 00000000 100cc220 1006b530 00000000
GPR24: 00000001 c0780d24 00029202 c0780e08 c0770000 00000000 c08578a4 00000063
NIP [c0348f64] sysrq_handle_crash+0x14/0x20
LR [c034974c] __handle_sysrq+0xcc/0x1d0
Call Trace:
[db18de70] [c0349734] __handle_sysrq+0xb4/0x1d0 (unreliable)
[db18dea0] [c03498ac] write_sysrq_trigger+0x5c/0x70
[db18deb0] [c01662f0] proc_reg_write+0x80/0xc0
[db18dee0] [c010ec50] vfs_write+0xc0/0x170
[db18df00] [c010ee80] sys_write+0x50/0x110
[db18df40] [c0011d58] ret_from_syscall+0x0/0x4
--- Exception: c01 at 0xff16ba4
LR = 0xfebad5c
Instruction dump:
39600003 7d605f9e 556b103a 7c005b78 98090003 4e800020 60000000 38000001
3d20c07a 90092170 7c0004ac 39200000 <98090000> 4e800020 60000000 3803ffd0
Sending IPI to other cpus...
Bye!
Using P2041 RDB machine description
......
/**********************/
Best Regards,
James
Liu Jianbo | WIND RIVER | Senior Engineer - Technical Support
Tel 86 28 65318098 | Cell 86 13558641588 | Fax 86 28 65319983
Manage your support account:
https://support.windriver.com
Ask a Technical Question
https://ask.windriver.com
Submit a Service Request
https://windriver.force.com/support
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
Hello James,
crash and kgdb are two completely different debuggers to diagnose the
kernel
panic issues. You can not access or start the crash from the context of
kgdb.
[**crash utility:**]
There are two ways to invoke crash utility[[1]];
1) Typical postmortem debugging: (after panic)
# crash /path/to/vmcore /path/to/vmlinux
o Kernel object file and memory image are supplied, respectively.
2) Live memory debugging:
# crash
o Pre-defined directories are searched for proper vmlinux
o Version string matched to the running kernel (/proc/version)
**OR**
# crash /path/to/vmlinux
[**kdb/kgdb:**][[2]]
1) You can put the target system in debug mode using SysRq event (g).
# echo g > /proc/sysrq-trigger
Eg:
# echo g > /proc/sysrq-trigger
[ 181.300854] SysRq : DEBUG
Entering kdb (current=0xffff8800766ebc60, pid 2347) on processor 1 due to
Keyboard Entry
[1]kdb> summary
sysname Linux
release 3.10.84
version #1 SMP Tue Jul 28 19:53:37 IST 2015
machine x86_64
nodename localhost.localdomain
domainname (none)
ccversion CCVERSION
date 2015-12-30 13:51:06 tz_minuteswest 0
uptime 00:03
load avg 0.11 0.11 0.04
MemTotal: 2050340 kB
MemFree: 1707332 kB
Buffers: 764 kB
1]kdb> dmesg | grep DMI:
<7>[ 0.000000] DMI: Red Hat KVM, BIOS 0.5.1 01/01/2007
[1]kdb> bt
Stack traceback for pid 2347
0xffff8800766ebc60 2347 624 1 1 R 0xffff8800766ec240 *bash
ffff88007bd61e70 0000000000000018
Call Trace:
<#DB> <<EOE>> [<ffffffff810dd84c>] ?
sysrq_handle_dbg+0x2c/0x50
[<ffffffff8134b312>] ? __handle_sysrq+0xa2/0x170
[<ffffffff8134b80a>] ? write_sysrq_trigger+0x4a/0x50
[<ffffffff811fb61d>] ? proc_reg_write+0x3d/0x80
[<ffffffff811993bd>] ? vfs_write+0xbd/0x1e0
[<ffffffff81199d89>] ? SyS_write+0x49/0xa0
[<ffffffff815a27c7>] ? tracesys+0xdd/0xe2
[1]kdb> rd
ax: 0000000000000001 bx: ffffffff818a4560 cx: ffff88007fd0eec0
dx: 0000000000000000 si: 0000000000000000 di: 0000000000000067
bp: ffff88007bd61e70 sp: ffff88007bd61e70 r8: ffffffff81b9069c
r9: 0000000000000248 r10: 0000000000000247 r11: 0000000000000003
r12: 0000000000000067 r13: 0000000000000246 r14: 0000000000000007
r15: 0000000000000000 ip: ffffffff810dd7f4 flags: 00000002 cs: 00000010
ss: 00000018 ds: 00000018 es: 00000018 fs: 00000018 gs: 00000018
[1]kdb> go
[ 251.553055] systemd-journald[2372]: File
/run/log/journal/71b46828837d4b1cb7f04385c86e7626/system.journal corrupted
or uncleanly shut down, renaming and replacing.
# <<<---{ root user }
[Note:] kdb command "go" is used to continue kernel execution. In above
case
it will take you back to bash shell.
2) You can crash the target system using SysRq event (c):
# echo c > /proc/sysrq-trigger
Eg:
# echo c > /proc/sysrq-trigger
[ 28.963227] SysRq : Trigger a crash
[ 28.964025] BUG: unable to handle kernel NULL pointer dereference
at (null)
[ 28.964025] IP: [<ffffffff8134abb6>] sysrq_handle_crash+0x16/0x20
[ 28.964025] Oops: 0002 [#1] SMP
Entering kdb (current=0xffff880077275a90, pid 2352) on processor 1 Oops:
(null)
due to oops @ 0xffffffff8134abb6
dCPU: 1 PID: 2352 Comm: bash Not tainted 3.10.84 #1
dHardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
dtask: ffff880077275a90 ti: ffff880076494000 task.ti: ffff880076494000
dRIP: 0010:[<ffffffff8134abb6>] [<ffffffff8134abb6>]
sysrq_handle_crash+0x16/0x20
dRSP: 0018:ffff880076495e80 EFLAGS: 00010096
dRAX: 000000000000000f RBX: ffffffff8190dbc0 RCX: ffff88007fd0eec0
dRDX: 0000000000000000 RSI: ffff88007fd0d368 RDI: 0000000000000063
dRBP: ffff880076495e80 R08: ffffffff81b9069c R09: 0000000000000245
dR10: 0000000000000244 R11: 0000000000000003 R12: 0000000000000063
dR13: 0000000000000246 R14: 0000000000000007 R15: 0000000000000000
dFS: 00007f53aec0c740(0000) GS:ffff88007fd00000(0000)
knlGS:0000000000000000
dCS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
dCR2: 0000000000000000 CR3: 00000000764bd000 CR4: 00000000000006e0
dDR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
dDR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
dStack:
ffff880076495eb8 ffffffff8134b312 0000000000000002 00007f53aec0a000
ffff880076495f50 0000000000000002 0000000000000000 ffff880076495ed8
ffffffff8134b80a 00007f53aec0a000 ffff88007661c000 ffff880076495ef8
dCall Trace:
more>
Only 'q' or 'Q' are processed at more prompt, input ignored
d [<ffffffff8134b312>] __handle_sysrq+0xa2/0x170
d [<ffffffff8134b80a>] write_sysrq_trigger+0x4a/0x50
d [<ffffffff811fb61d>] proc_reg_write+0x3d/0x80
d [<ffffffff811993bd>] vfs_write+0xbd/0x1e0
d [<ffffffff81199d89>] SyS_write+0x49/0xa0
d [<ffffffff815a27c7>] tracesys+0xdd/0xe2
dCode: eb 9b 45 01 f4 45 39 65 34 75 e5 4c 89 ef e8 02 f8 ff ff eb db 0f
1f 44 00 00 55 c7 05 20 17 54 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25
00 00 00 00 01 5d c3 0f 1f 44 00 00 55 31 c0 c7 05 7e
[1]kdb> summary
sysname Linux
release 3.10.84
version #1 SMP Tue Jul 28 19:53:37 IST 2015
machine x86_64
nodename localhost.localdomain
domainname (none)
ccversion CCVERSION
date 2015-12-30 14:07:49 tz_minuteswest 0
uptime 00:01
load avg 0.95 0.21 0.06
MemTotal: 2050340 kB
MemFree: 1775224 kB
Buffers: 764 kB
[1]kdb> bt
Stack traceback for pid 2352
0xffff880077275a90 2352 629 1 1 R 0xffff880077276070 *bash
ffff880076495e80 0000000000000018 ffff880076495eb8 ffffffff8134b312
0000000000000002 00007f53aec0a000 ffff880076495f50 0000000000000002
0000000000000000 ffff880076495ed8 ffffffff8134b80a 00007f53aec0a000
Call Trace:
[<ffffffff8134b312>] ? __handle_sysrq+0xa2/0x170
[<ffffffff8134b80a>] ? write_sysrq_trigger+0x4a/0x50
[<ffffffff811fb61d>] ? proc_reg_write+0x3d/0x80
[<ffffffff811993bd>] ? vfs_write+0xbd/0x1e0
[<ffffffff81199d89>] ? SyS_write+0x49/0xa0
[<ffffffff815a27c7>] ? tracesys+0xdd/0xe2
[1]kdb> rd
ax: 000000000000000f bx: ffffffff8190dbc0 cx: ffff88007fd0eec0
dx: 0000000000000000 si: 0000000000000000 di: 0000000000000063
bp: ffff880076495e80 sp: ffff880076495e80 r8: ffffffff81b9069c
r9: 0000000000000245 r10: 0000000000000244 r11: 0000000000000003
r12: 0000000000000063 r13: 0000000000000246 r14: 0000000000000007
r15: 0000000000000000 ip: ffffffff8134abb6 flags: 00010096 cs: 00000010
ss: 00000018 ds: 00000018 es: 00000018 fs: 00000018 gs: 00000018
[1]kdb> go
Catastrophic error detected
kdb_continue_catastrophic=0, type go a second time if you really want to
continue
[1]kdb> go
Catastrophic error detected
kdb_continue_catastrophic=0, attempting to continue
[ 28.964025] Modules linked in: ip6t_rpfilter ip6t_REJECT ipt_REJECT
xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter
ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6
ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables
iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat
nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter
snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm
snd_page_alloc snd_timer snd serio_raw pcspkr i2c_piix4 virtio_balloon
soundcore nfsd auth_rpcgss nfs_acl lockd sunrpc ip_tables xfs libcrc32c
ata_generic pata_acpi ata_piix cirrus syscopyarea sysfillrect sysimgblt
drm_kms_helper ttm virtio_net virtio_blk drm libata virtio_pci virtio_ring
virtio i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
[ 112.090263] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[ 112.090263] task: ffff880077275a90 ti: ffff880076494000 task.ti:
ffff880076494000
[ 112.090263] RIP: 0010:[<ffffffff8134abb6>] [<ffffffff8134abb6>]
sysrq_handle_crash+0x16/0x20
[ 112.090263] RSP: 0018:ffff880076495e80 EFLAGS: 00010096
[ 112.090263] RAX: 000000000000000f RBX: ffffffff8190dbc0 RCX:
ffff88007fd0eec0
[ 112.090263] RDX: 0000000000000000 RSI: ffff88007fd0d368 RDI:
0000000000000063
[ 112.090263] RBP: ffff880076495e80 R08: ffffffff81b9069c R09:
0000000000000245
[ 112.090263] R10: 0000000000000244 R11: 0000000000000003 R12:
0000000000000063
[ 112.090263] R13: 0000000000000246 R14: 0000000000000007 R15:
0000000000000000
[ 112.090263] FS: 00007f53aec0c740(0000) GS:ffff88007fd00000(0000)
knlGS:0000000000000000
[ 112.090263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 112.090263] CR2: 0000000000000000 CR3: 00000000764bd000 CR4:
00000000000006e0
[ 112.090263] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 112.104034] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 112.104034] Stack:
[ 112.104034] ffff880076495eb8 ffffffff8134b312 0000000000000002
00007f53aec0a000
[ 112.104034] ffff880076495f50 0000000000000002 0000000000000000
ffff880076495ed8
[ 112.104034] ffffffff8134b80a 00007f53aec0a000 ffff88007661c000
ffff880076495ef8
[ 112.104034] Call Trace:
[ 112.104034] [<ffffffff8134b312>] __handle_sysrq+0xa2/0x170
[ 112.104034] [<ffffffff8134b80a>] write_sysrq_trigger+0x4a/0x50
[ 112.104034] [<ffffffff811fb61d>] proc_reg_write+0x3d/0x80
[ 112.104034] [<ffffffff811993bd>] vfs_write+0xbd/0x1e0
[ 112.104034] [<ffffffff81199d89>] SyS_write+0x49/0xa0
[ 112.104034] [<ffffffff815a27c7>] tracesys+0xdd/0xe2
[ 112.104034] Code: eb 9b 45 01 f4 45 39 65 34 75 e5 4c 89 ef e8 02 f8 ff
ff eb db 0f 1f 44 00 00 55 c7 05 20 17 54 00 01 00 00 00 48 89 e5 0f ae f8
<c6> 04 25 00 00 00 00 01 5d c3 0f 1f 44 00 00 55 31 c0 c7 05 7e
[ 112.104034] RIP [<ffffffff8134abb6>] sysrq_handle_crash+0x16/0x20
[ 112.104034] RSP <ffff880076495e80>
[ 112.104034] CR2: 0000000000000000
[ 112.104034] ---[ end trace 4484e44d0167e21c ]---
[ 112.104034] Kernel panic - not syncing: Fatal exception
PANIC: Fatal exception
Entering kdb (current=0xffff880077275a90, pid 2352) on processor 1 due to
Keyboard Entry
[Note:] kdb command "go" is used to continue kernel execution. In above
case
it will not take you back to bash shell. :)
Few questions for you;
[Q:1] What exactly you are trying to achieve ?
[Q:2] What is the exact issue ?
Regards,
BKS
[[1]] http://people.redhat.com/anderson/crash_whitepaper/#INVOCATION
[[2]] https://www.kernel.org/pub/linux/kernel/people/jwessel/kdb/
3254
days inactive
3254
days old
devel@lists.crash-utility.osci.io
1 comments
2 participants
participants (2)
-
Buland Kumar Singh -
Liu, Jianbo (James)