Re: [Crash-utility] PATCH 00/10] teach crash to work with "live" ramdump
----- Original Message -----
Hi Dave,
Recently I used crash-tool for the first time and I was pleasantly surprised,
it really looks like a very useful and handy debugging tool ;)
And I was surprised again when I figured out that it can be used to debug the
live system on the same machine. Cool!
Now I am wondering if we can teach it to debug the live guests runnning under
qemu/kvm. This looks certainly possible, qemu supports gdb remote protocol.
But. this obviously needs more work. And. Afaics crash-tool needs some fixes
anyway. See 01/10-07/10, but probably it needs more changes, so far I only
tried to audit task.c and kernel.c. I tried to document every change, but I
am very new to this code so I can be easily wrong.
So can't we teach it to work with live/raw RAM dumpfiles for the start? See
09/10 and 10/10.
With these changes I can run qemu-kvm with the
-object memory-backend-file,id=MEM,size=128m,mem-path=/tmp/MEM,share=on
-numa node,memdev=MEM
options and then do
$ crash path-to-guests-vmlinux live:/tmp/MEM@0
to debug the live guest, No need to dump-guest-memory + restart
/usr/bin/crash
which can be slow.
What do you think?
Oleg.
defs.h | 1 +
filesys.c | 8 ++++----
kernel.c | 7 +++----
main.c | 11 +++++++++++
ramdump.c | 49 ++++++++++++++++++++++++++++++++++---------------
task.c | 13 ++++++-------
tools.c | 2 +-
7 files changed, 60 insertions(+), 31 deletions(-)
Hi Oleg,
This is going to take some time for me to digest. But without my studying it
at all, there are a couple things that immediately come to mind.
First, a couple of nits about the files being modified.
I see that your overloading the ramdump.c file, but the ramdump facility was
proposed and added by companies that use a firmware-based facility for dumping
raw ARM/ARM64/MIPS RAM into one or more ramdump files, typically because they
do/did not have kdump capability with their embedded hardware. I really don't
feel comfortable conflating that facility with a remote/live QEMU/KVM memory
source. Those companies support it, and so any changes that you've added to
ramdump.c should be accomplished elsewhere.
Also, we don't want to get this confused in any way with the REMOTE_xxx
stuff. The remote.c file and remote access facility was deprecated many
years ago, but recently resurrected in crash-7.0.4 by Verizon specifically
for use like so:
- Resurrection of the remote analysis capability for use with the
"xen-crashd" daemon running on a Xen Dom0 host, which communicates
with a paused or shutdown DomU guest kernel. The daemon can be
accessed like so:
$ crash localhost:5001,/dev/xenmem vmlinux
(dslutz(a)verizon.com
I don't think you are modifying their "resurrected" usage, but just an FYI
since
you refer to it in your patches.
From a crash utility perspective, this QEMU/KVM live memory source is
a completely
new live memory source, and as such should have its own source file
and/or set of
functions, as is the case with the other actual dumpfile types, or live memory
sources /proc/kcore, /dev/mem and /dev/crash.
Now, more to the point...
Anyway, this is a great idea, and one I've pondered in the past, but is there a
concrete reason that it could not be a simple matter of plugging in a new function
into pc->readmem()? Currently there are 20+ pluggable readmem functions that exist
for supporting all of the supported dumpfile types and live memory sources.
Regardless of how it's accomplished, the plug-in basically just needs to be able to
take a physical address and a count (all guaranteed to be contained within a page),
and access/return it. How it accomplishes that task is hidden from the mainstream
crash code.
Thanks,
Dave