Hi Dave, this is excatly I wrote in previous mail....have ELF header and one program header. I do have modified version of crash utlity (it is quiet a bit of code)  (introduced ramdump.c) I feel it is better to ahve that feature integrated into crash utility for raw ramdump. Regards, Oza. On Tuesday, 27 May 2014 7:05 PM, Dave Anderson <anderson(a)redhat.com&gt; wrote: ----- Original Message ----- I'm presuming that these binary files are generated from a 32-bit ARM machine? Anyway, if it's just a raw dump of physical memory, the crash utility cannot do anything with them.  Crash supports over 20 different dumpfile formats, but they all have dumpfile headers that describe what's contained in the file.  It seems to me that there might have been a discussion on this list a while ago about the pre-pending of an ELF header to a raw, memory-only, ARM dump?  Perhaps the ARM users on this list have some recollection? Dave -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

----- Original Message ----- Since these raw DDR dumps seem to be an existing feature, it certainly seems worth implementing support for them. I'm curious as to how these RAM dumps are currently used -- are there other tools that use them somehow? Are these DDR RAM dumps specific to embedded 32-bit ARM machines? Your feature sounds like a two-stage process: (1) invoke crash utility -- passing it the base physical address of the contiguous RAM dump, and the RAM dump file name(s) -- and then crash creates a single ELF vmcore by pre-pending an ELF header and concatenating the dump file names. (2) invoke crash utility with vmlinux and newly-created vmcore. But you mention "generate object files" above. Do you generate more than one file? Is the newly-created vmcore subsequently recognized as a netdump or kdump ELF vmcore? (i.e, handled by existing code in netdump.c) Or does it create a new ELF-like dumpfile type that is handled in your new ramdump.c file? Could it be done in one step? In other words, something like: $ crash vmlinux --ddr 80000000 ddr1.bin [ddr2.bin ...] where there would be a "virtual" ELF header created that could be used during the crash session? (perhaps with an optional "-o outputfile" command line option to create/save it as an ELF vmcore) Dave

Typo Like kdump and merino lets call this one more format raw_ramdump_elf32 Like kdump and netdump lets call this one more format raw_ramdump_elf32 Sent from Yahoo Mail on Android

Hi Dave, please find all my comments/clarifications/answers inline below. Since these raw DDR dumps seem to be an existing feature, it certainly seems worth implementing support for them.   I'm curious as to how these RAM dumps are currently used -- are there other tools that use them somehow?  Oza: there could be various tools which organisation might be writing to extract some logs, because they know the physical address where their private/public logs are stored. but none of the tools match the ability of crash utility as we all agree. Are these DDR RAM dumps specific to embedded 32-bit ARM machines? Oza: yes it depends, if it  is taken on armv7 then 32bit, if it is armv8, then 64 bit.   Your feature sounds like a two-stage process: (1) invoke crash utility -- passing it the base physical address of the     contiguous RAM dump, and the RAM dump file name(s) -- and then crash     creates a single ELF vmcore by pre-pending an ELF header and concatenating     the dump file names. (2) invoke crash utility with vmlinux and newly-created vmcore. Oza: yes it is 2 stage process, in the first stage crash utility will prepare something which crash utility itself can understand. which is nothing but ELF format, which every debugger typically understands (such as gdb) so I do that conversion at the first place. once that conversion is in place, crash utility core code takes it over and do the rest of the job. But you mention "generate object files" above.  Do you generate more than one file? Oza: what I meant by object file is ELF file (ELF is something which is ready to be loaded/linked) this object file is nothing but one temporary file which crash utility will create... so if you give crash ./vmlinux ./ramdump.bin kernel_base_address it woudl have created ramdump_ARM_ELF32  (temporary file) which crash utility will understand). after you exit crash, the temporary filw would be deleted. Is the newly-created vmcore subsequently recognized as a netdump or kdump ELF vmcore?  (i.e, handled by existing code in netdump.c)  Oza: it would be recognized as kdump (because I am currently using 'flags' of kdump), thsi is just to eb aligned with crash internal implementations. but nontheless it is a separate ELF file Or does it create a new ELF-like dumpfile type that is handled in your new ramdump.c file? Oza: it is separate ELF file. Could it be done in one step?  In other words, something like:   $ crash vmlinux --ddr 80000000 ddr1.bin [ddr2.bin ...] Oza: this is precisely what is hapepning. where there would be a "virtual" ELF header created that could be used during the crash session?  (perhaps with an optional "-o outputfile" command line option to create/save it as an ELF vmcore)  Oza: virtual ELF header, and one more virtual program header would be created in temporary file which, the budle, called ELF32. optional command line option could be created as well. but currently we take bin file only. please feel free to ask anymore questions. Oza. On Wednesday, 28 May 2014 9:50 PM, Dave Anderson <anderson(a)redhat.com&gt; wrote: ----- Original Message ----- Yes -- please answer all of the questions in my email. What is "merino"? Dave

Hi Dave, 1) what I talked about is pure 32bit (not with PAE, but should not be a problem with PAE) 64 bit ELF headers could also be in place. but my first and important point is: get 32bit implementation right (rest will fall after that) in other words, the design/framework has to be made in such a way that existing crash utility absorbs both ELF32 and 64 and even any other debugging format or not to mention for long term but 128 bit ELF format as well :) kdump should be capable of reading it I think. 2) OK, so then after you've created the temporary ELF file, crash internally would end up setting the pc->readmem() pointers to read_kdump()?  Or do you have your own readmem() function in the new ramdump.c file? currently I am setting function point to kdump, so I dont have separate readmem function. 3) Clarify one more thing please: is the temporary file simply a little standalone ELF header that describes what's in the ddr.bin file?  Or do you concatenate the ELF header and the ddr.bin file into a single, temporary, ELF vmcore file that is essentially a clone of an ELF kdump dumpfile? it looks more closer to the former one which you mentioned. standalone ELF object. (Vmcore is ELF format anyway, so should be similar) not sure about kdump format though. PS: It might take some time to post the patch because I wrote it in hurry to make it work at the first point:but now I have to think with the design perspective and more on long term perspective to make this framework scalable and get easily accommodated in crash utility architecture. Regards, Oza. On Thursday, 29 May 2014 7:09 PM, Dave Anderson <anderson(a)redhat.com&gt; wrote: ----- Original Message ----- Are you thinking of supporting this feature for ARM64 as well? Also, by any chance are you working with 32-bit ARM with PAE?  Currently there is no support in the crash utility for ARM/PAE, and I have been hoping/waiting for ARM developers on this list to develop and post patches supporting it.  Although it's unclear whether the kdump facility would even work as-is with PAE, if this feature were available, it would be an an alternative to kdump.  Anyway, if/when ARM/PAE support is put in place, it would require the usage of 64-bit ELF headers for the 32-bit ARM/PAE dumps. But I'm getting ahead of myself...  ;-) OK, so then after you've created the temporary ELF file, crash internally would end up setting the pc->readmem() pointers to read_kdump()?  Or do you have your own readmem() function in the new ramdump.c file? Clarify one more thing please: is the temporary file simply a little standalone ELF header that describes what's in the ddr.bin file?  Or do you concatenate the ELF header and the ddr.bin file into a single, temporary, ELF vmcore file that is essentially a clone of an ELF kdump dumpfile? Also, I suggested the "-o outputfile" option -- after which you could simply enter "crash vmlinux outputfile" -- because your implementation requires that the user must be aware of the base physical address value.  Anyway, this all sounds good, so please post a patch.  And can you also make one of these dumpfiles available to me to download?  Thanks,   Dave

----- Original Message ----- If /dev/mem is used as the live memory source, then it is possible to modify memory. It's theoretically possible to use it with ELF dumpfiles, but the vmcore is opened O_RDONLY and cmd_wr() doesn't allow it. Right -- and then the decision needs to be made whether it's necessary to create a special dumpfile type, or simply create a kdump ELF clone. More on that below... You could continue to use the KDUMP flag given that you are essentially creating a clone of a kdump ELF vmcore. That is what is done with KVM guest vmcores that are created with "virsh dump --memory-only". The KDUMP flag is set, but because that facility also creates an extra ELF note with the name string "QEMU", the QEMU_MEM_DUMP flag is set in pc->flags2. That extra flag could be used if a differentiator is needed. You could do the same kind of thing. On the other hand, if you do think its best to create a new dumpfile-type flag, then some existing pc->flags #define (which is currently full) needs to be moved to pc->flags2, and your new flag could take its place. That's necessary because the MEMORY_SOURCES and DUMPFILE_TYPES require that their bit-flags contents are in pc->flags. Unless you have a compelling need for it, the write function be a placeholder that just returns WRITE_ERROR. Or just don't bother setting pc->writemem during initialization -- it will never get called anyway. Thanks, Dave

----- Original Message ----- I haven't tested this yet, but I do note that the patch doesn't apply because it appears like you were working with crash-7.0.5? In the future, please post patches against either the upstream git development branch, or at least against the current upstream release, currently crash-7.0.7. Anyway, before going any further, the whole command line user-interface is too complex, verbose, and error-prone. Also, the patch adds too much RAM-dump related stuff to main(), where the "nodes", "rams", "rd" and "elf_out" variables really belong in the ramdump.c file. This is really just another dumpfile type, and it should be handled in a similar manner as the other types. Now, breaking this down, what we have to handle during invocation in main() is one or more blocks of RAM that start at some physical address, i.e., one or more "ram@address" ordered pairs. Why not have a single, uniquely-identifiable argument string that describes the whole entity? Here is my counter-proposal, where the invocation would be something like this: $ crash vmlinux ramdump@address [-o output_file] and if there are multiple images, make them a comma-separated list: $ crash vmlinux ramdump1@address,ramdump2@address [-o output_file] Then here in main(), handle it like the other "is_dumpfile_type()" functions by passing the single argument string to an is_ramdump() function in your ramdump.c file: while (argv[optind]) { + if (is_ramdump(argv[optind]) { + if (pc->flags & MEMORY_SOURCES) { + error(INFO, "too many dumpfile arguments\n"); + program_usage(SHORT_FORM); + } + pc->dumpfile = ramdump_to_elf(); + if (is_kdump(pc->dumpfile, KDUMP_LOCAL)) { + pc->flags |= KDUMP; + pc->readmem = read_kdump; + pc->writemem = write_kdump; + } else { + error(INFO, "malformed ELF dump file: %s\n", + pc->dumpfile); + program_usage(SHORT_FORM); + } + continue; + } if (is_remote_daemon(argv[optind])) { if (pc->flags & DUMPFILE_TYPES) { error(INFO, "too many dumpfile/memory arguments\n"); program_usage(SHORT_FORM); } pc->flags2 |= REMOTE_DAEMON; optind++; continue; } The is_ramdump() function can verify if it's a valid argument by first checking for the required/unique "@" character in the argument string. It can then parse the argument, and save the required "nodes", "rams", etc. in a "ramdump_def" structure that can be statically declared in ramdump.c. You should also have a dump_ramdump_def() function that can be called from "help -[Nd]". With respect to the [-o output_file], and given the potential simplicity of the argument string, I think it should be optional. You could do something like this in the getopt() handler, and have the ELF output_file name pre-stored in the ramdump_def structure: + case 'o': + ramdump_elf_output_file(optarg); + break; If "-o output_file" is NOT used, then ramdump_to_elf() can pass back the name of a temporary file. Great, thanks, I've got them... I'd also really appreciate having a sample RAM dump from an ARM64 machine as well. Thanks, Dave

On Wed, Jul 23, 2014 at 6:37 PM, Dave Anderson <anderson(a)redhat.com&gt; wrote: Modified patch attached. It is rebased to latest crash version. The arguments are in the form of ordered pair as you had mentioned. I have tested it with arm and armv8 ramdumps. Do we really need dump_ramdump_def ? As the dump is converted to kdump and we use the kdump flag in pc->flags, help -D and help -n works fine using kdump dump functions. Did I miss something ? I will send you the link to arm64 ramdump in another email. Thanks, Vinayak

----- Original Message ----- I tested your latest patch on the sample ARM and ARM64 RAM dumps you sent me. As far as the patch itself is concerned, I ran into a problem where if crash is invoked in a directory where it does not have write permission, the session hangs trying to write to a bad file descriptor -- because of this: fd2 = open(out_elf, O_CREAT|O_RDWR, S_IRUSR|S_IWUSR); if (!fd2) { error(INFO, "%s open error\n", out_elf); goto end1; } It should be "if (fd2 < 0)". But more to the point, in my earlier response, I had suggested this: I should have been more clear w/respect to "a temporary file". what I was suggesting was that you do something like using mkstemp(3) to create a temporay file in /var/tmp, and then unlink() it immediately so it would only exist until the crash session ends. I tested this patch on your sample ARM and ARM64 RAM dumps. The 32-bit ARM dumpfile can be analyzed OK, but as you noted, the ARM64 dump requires "--cpus 4" to come OK, which really should not be required. Investigating the reason for the "--cpus 4" requirement, it's helpful to compare the two dumps. With your sample 32-bit ARM dumpfile, although it comes up OK with 4 cpus, note that only cpu 1 is marked online in the kernel: crash> help -k ... cpu_possible_map: 0 1 2 3 cpu_present_map: 0 1 2 3 cpu_online_map: 1 cpu_active_map: 0 1 2 3 ... The 32-bit arm.c arm_get_smp_cpus() function calculates the number of cpus like this: return MAX(get_cpus_active(), get_cpus_online()); so it returns 4 since the "active" map shows all 4 cpus. The "ps" command shows tasks associated with all 4 cpus, and the runqueues look like this, where cpus 0 and 2 have their idle task running, and cpus 1 and 3 have user-mode tasks running: crash> runq CPU 0 RUNQUEUE: c0f286c0 CURRENT: PID: 0 TASK: c0a5d8b0 COMMAND: "swapper/0" RT PRIO_ARRAY: c0f287a0 [no tasks queued] CFS RB_ROOT: c0f28730 [no tasks queued] CPU 1 RUNQUEUE: c0f316c0 CURRENT: PID: 13429 TASK: db944580 COMMAND: "AudioIn_5F8" RT PRIO_ARRAY: c0f317a0 [no tasks queued] CFS RB_ROOT: c0f31730 [120] PID: 474 TASK: d9a36ac0 COMMAND: "kworker/1:1" [120] PID: 2890 TASK: c89b2580 COMMAND: "sh" CPU 2 RUNQUEUE: c0f3a6c0 CURRENT: PID: 0 TASK: db63a040 COMMAND: "swapper/2" RT PRIO_ARRAY: c0f3a7a0 [no tasks queued] CFS RB_ROOT: c0f3a730 [112] PID: 1599 TASK: db87d040 COMMAND: "mm_device_threa" CPU 3 RUNQUEUE: c0f436c0 CURRENT: PID: 1949 TASK: db951040 COMMAND: "WindowManager" RT PRIO_ARRAY: c0f437a0 [no tasks queued] CFS RB_ROOT: c0f43730 [no tasks queued] crash> So it does seem that whatever mechanism you use to take the raw RAM dump on the 32-bit ARM offlines cpus first? Now, on the ARM64 dumpfile, if I force it to come up with "--cpus 4" it shows that only cpu 0 is online, present and active: crash> help -k ... cpu_possible_map: 0 1 2 3 cpu_present_map: 0 cpu_online_map: 0 cpu_active_map: 0 ... I can understand that perhaps cpus are offlined prior to taking the RAM dump, but it's strange that the "present" and "active" maps are also the same as the "online" map? Currently the arm64.c arm64_get_smp_cpus() returns the number of cpus like this: return MAX(get_cpus_online(), get_highest_cpu_online()+1); so it returns 1. Even if it did it the same as the 32-bit ARM, it would still return 1 because of the active map. So we have to force it to return 4 with "--cpus 4". But having done that, oddly enough, the "runq" command shows this, where the "CURRENT" task on cpu 0 is "0": crash> runq CPU 0 RUNQUEUE: ffffffc03ffb6e40 CURRENT: 0 RT PRIO_ARRAY: ffffffc03ffb6fb0 [no tasks queued] CFS RB_ROOT: ffffffc03ffb6f10 [no tasks queued] CPU 1 RUNQUEUE: ffffffc03ffc1e40 CURRENT: PID: 0 TASK: ffffffc03ecb4b00 COMMAND: "swapper/1" RT PRIO_ARRAY: ffffffc03ffc1fb0 [no tasks queued] CFS RB_ROOT: ffffffc03ffc1f10 [no tasks queued] CPU 2 RUNQUEUE: ffffffc03ffcce40 CURRENT: PID: 0 TASK: ffffffc03ecb5dc0 COMMAND: "swapper/2" RT PRIO_ARRAY: ffffffc03ffccfb0 [no tasks queued] CFS RB_ROOT: ffffffc03ffccf10 [no tasks queued] CPU 3 RUNQUEUE: ffffffc03ffd7e40 CURRENT: PID: 0 TASK: ffffffc03ecf0000 COMMAND: "swapper/3" RT PRIO_ARRAY: ffffffc03ffd7fb0 [no tasks queued] CFS RB_ROOT: ffffffc03ffd7f10 [no tasks queued] crash> I have never seen this before -- As I understand it, if no other task is queued and run on a cpu, then it defaults to the idle/swapper task for that cpu, whose address is hard-wired in the per-cpu runqueue structure. But if I look at the rq structure for cpu 0, not only is the "curr" task pointer NULL, the "idle" task pointer is also: crash> rq.curr,idle,cpu ffffffc03ffb6e40 curr = 0x0 idle = 0x0 cpu = 0 crash> whereas the other 3 cpus show that they are running their idle tasks: crash> rq.curr,idle,cpu ffffffc03ffc1e40 curr = 0xffffffc03ecb4b00 idle = 0xffffffc03ecb4b00 cpu = 1 crash> rq.curr,idle,cpu ffffffc03ffcce40 curr = 0xffffffc03ecb5dc0 idle = 0xffffffc03ecb5dc0 cpu = 2 crash> rq.curr,idle,cpu ffffffc03ffd7e40 curr = 0xffffffc03ecf0000 idle = 0xffffffc03ecf0000 cpu = 3 crash> Perhaps it has something to do with *when* you took the dump. The "sys" command shows an UPTIME of 00:00:00: crash> sys KERNEL: /home/anderson/Downloads/tmp_ARM64/vmlinux DUMPFILE: ramdump_elf CPUS: 4 DATE: Wed Dec 31 19:00:00 1969 UPTIME: 00:00:00 LOAD AVERAGE: 0.00, 0.00, 0.00 TASKS: 34 NODENAME: (none) RELEASE: 3.10.33+ VERSION: #22 SMP PREEMPT Tue May 6 16:23:34 IST 2014 MACHINE: aarch64 (unknown Mhz) MEMORY: 1 GB PANIC: "" crash> And the "ps" command doesn't show any user-space tasks running, not even "init" PID 1, and the funky idle/swapper task on cpu 0 shows a PID of 1: crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM 1 -1 0 ffffffc03ec78000 UN 0.0 0 0 [swapper/0] 2 -1 0 ffffffc03ec792c0 IN 0.0 0 0 [kthreadd] 3 2 0 ffffffc03ec7a580 IN 0.0 0 0 [ksoftirqd/0] 4 2 0 ffffffc03ec7b840 IN 0.0 0 0 [kworker/0:0] 5 2 0 ffffffc03ec7cb00 IN 0.0 0 0 [kworker/0:0H] 6 2 0 ffffffc03ec7ddc0 IN 0.0 0 0 [kworker/u8:0] 7 2 0 ffffffc03ecb0000 IN 0.0 0 0 [migration/0] 8 2 0 ffffffc03ecb12c0 IN 0.0 0 0 [rcu_preempt] 9 2 0 ffffffc03ecb2580 IN 0.0 0 0 [rcu_bh] 10 2 0 ffffffc03ecb3840 IN 0.0 0 0 [rcu_sched] 11 2 1 ffffffc03ecf12c0 ?? 0.0 0 0 [migration/1] 12 2 1 ffffffc03ecf2580 ?? 0.0 0 0 [ksoftirqd/1] 13 2 1 ffffffc03ecf3840 IN 0.0 0 0 [kworker/1:0] 14 2 1 ffffffc03ecf4b00 IN 0.0 0 0 [kworker/1:0H] 15 2 2 ffffffc03ecf5dc0 ?? 0.0 0 0 [migration/2] 16 2 2 ffffffc03ed20000 IN 0.0 0 0 [ksoftirqd/2] 17 2 0 ffffffc03ed212c0 UN 0.0 0 0 [kworker/2:0] 18 2 2 ffffffc03ed22580 IN 0.0 0 0 [kworker/2:0H] 19 2 3 ffffffc03ed23840 IN 0.0 0 0 [migration/3] 20 2 3 ffffffc03ed24b00 IN 0.0 0 0 [ksoftirqd/3] 21 2 3 ffffffc03ed25dc0 IN 0.0 0 0 [kworker/3:0] 22 2 3 ffffffc03ed40000 IN 0.0 0 0 [kworker/3:0H] 23 2 0 ffffffc03ed412c0 IN 0.0 0 0 [khelper] 24 2 0 ffffffc03ed42580 IN 0.0 0 0 [kdevtmpfs] 25 2 0 ffffffc03ed43840 IN 0.0 0 0 [kworker/u8:1] 56 2 0 ffffffc03ededdc0 IN 0.0 0 0 [bcm_ipc_ch0] 57 2 0 ffffffc03edecb00 IN 0.0 0 0 [bcm_ipc_ch11] 180 2 0 ffffffc03ee5ddc0 IN 0.0 0 0 [writeback] 182 2 0 ffffffc03ee912c0 IN 0.0 0 0 [bioset] 184 2 0 ffffffc03ede92c0 IN 0.0 0 0 [kworker/u9:0] 185 2 0 ffffffc03ede8000 IN 0.0 0 0 [kblockd] crash> So I'm guessing that this dumpfile was taken before the "init" task was even created, and the kernel data structures were not fully initialized? Maybe you can try taking a RAM dump on an ARM64 machine after it is up and running? Thanks, Dave

----- Original Message ----- Testing with just the 32-bit ARM RAM dumpfile you gave me yielded some interesting, albeit somewhat strange, results. Running exactly as you specified looks like this: $ crash vmlinux AP__0x81e00000.hex@0x81e00000 ... [ cut ] ... DUMPFILE: /var/tmp/ramdump_elf2z0YjA ... [ cut ] ... crash> !readelf -a /var/tmp/ramdump_elf2z0YjA ELF Header: Magic: 7f 45 4c 46 01 01 01 03 00 00 00 00 00 00 00 00 Class: ELF32 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - GNU ABI Version: 0 Type: CORE (Core file) Machine: ARM Version: 0x1 Entry point address: 0x0 Start of program headers: 52 (bytes into file) Start of section headers: 0 (bytes into file) Flags: 0x0 Size of this header: 52 (bytes) Size of program headers: 32 (bytes) Number of program headers: 2 Size of section headers: 0 (bytes) Number of section headers: 0 Section header string table index: 0 There are no sections in this file. There are no sections to group in this file. Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align NOTE 0x000074 0x00000074 0x00000000 0x00000 0x00000 0 LOAD 0x000074 0x00000000 0x81e00000 0x1e200000 0x1e200000 RWE 0 There is no dynamic section in this file. There are no relocations in this file. No version information found in this file. crash> help -m | grep phys_base phys_base: 81e00000 crash> Everything looks good, and both the "PhysAddr" field in the PT_LOAD segment and the machdep->phys_base address correctly reflect the offset value. However, throughout the crash utility, if it is possible to non-ambiguously enter a hexadecimal address, you can enter the number without the "0x" prepended. And elsewhere, it just requires a hexadecimal value -- which should be the case with these dumpfile offsets. So I tried using "AP__0x81e00000.hex@81e00000" on the crash command line -- and although it seemed "to work", check out the PT_LOAD PhysAddr and the phys_base value: $ crash vmlinux AP__0x81e00000.hex@81e00000 ... [ cut ] ... crash> !readelf -a /var/tmp/ramdump_elfuDqyZq ... [ cut ] ... Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align NOTE 0x000074 0x00000074 0x00000000 0x00000 0x00000 0 LOAD 0x000074 0x00000000 0x00000051 0x1e200000 0x1e200000 RWE 0 ... crash> help -m | grep phys_base phys_base: 51 crash> That's because you're using strtoul(), which presumed that 81e00000 was a decimal string, threw out the e00000, used just the "81", and came up with the 0x51 value. And as it turns out, you can put just about anything as the physical offset: $ crash vmlinux AP__0x81e00000.hex@0xdeadbeef ... [ cut ] ... Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align NOTE 0x000074 0x00000074 0x00000000 0x00000 0x00000 0 LOAD 0x000074 0x00000000 0xdeadbeef 0x1e200000 0x1e200000 RWE 0 ... crash> help -m | grep phys_base phys_base: deadbeef crash> Including 0: $ crash vmlinux AP__0x81e00000.hex@0 ... [ cut ] ... Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align NOTE 0x000074 0x00000074 0x00000000 0x00000 0x00000 0 LOAD 0x000074 0x00000000 0x00000000 0x1e200000 0x1e200000 RWE 0 ... crash> help -m | grep phys_base phys_base: 0 crash> However, although it *appears* to work correctly, it fails to translate/read user-space or vmalloc addresses, because the PTEs contain reference to physical addresses that are based at 0x81e00000. If your dumpfile sample had been a kernel that was running with any kernel modules, you would have seen an error during initialization when the linked list of kernel modules gets read. In any case, if you use "htol()" instead of strtoul(), the user won't have to enter the "0x", because it will presume it's a hexadecimal address. However, you cannot put *nothing* as the offset value: $ crash vmlinux AP__0x81e00000.hex@ crash 7.0.8rc12 Copyright (C) 2002-2014 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. Segmentation fault (core dumped) $ So that also needs fixing. A few other questions/comments... When 32-bit x86 kdumps are created, they typically default to using the 64-bit ELF format. That is required in case there is physical memory above the 4GB threshold, which cannot be described in a 32-bit ELF header. Since 32-bit ARM can also be PAE, would it make more sense to *always* use 64-bit ELF headers for both ARM and ARM64? I don't see why not, and it should simplify ramdump.c quite a bit. And thinking this through a bit more, to me it seems really wasteful to have to create a whole new dumpfile. Even if it's only a temporary file, it's still seemingly an unnecessary duplication of disk space. Here's an idea, not fully thought through, but seems like it could work when the "temporary" dumpfile option is used: (1) Create a temporary file that *only* consists of the ELF header. (2) Set a new RAMDUMP flag in pc->flags2. (3) Pass that temporary ELF header file to is_kdump() as you do now. (4) is_kdump() passes it to is_netdump(), and I believe that is_netdump() should parse just the ELF header and accept it as a KDUMP_ELF32 or KDUMP_ELF64 file without even being aware that there's no physical memory data attached. (5) When kdump_init() is called, it passes through to netdump_init() which calls check_dumpfile_size() -- which would need to look at the RAMDUMP flag to do the right thing. (6) And instead of using read_kdump(), copy it to a new read_ramdump() function that reads from the original RAM dump image(s), figuring out the file offsets accordingly. Keep the new read_ramdump() function in netdump.c so it can continue to use the "nd" vmcore_data structure that is statically defined there. read_ramdump() may have to interact with function(s) in ramdump.c, or perhaps the ramdump_def structure can be shared with netdump.c somehow. There will presumably be a few other glitches that will require checking the new RAMDUMP flag, but I don't think that there would be anything that couldn't be overcome. That really would be the ideal way to handle these files. If the "-o elf_file" option is used, then you could proceed as you do now -- create the new full ELF file, and pass it to is_kdump() without having set the new RAMDUMP flag. What do you think? Dave

----- Original Message ----- Hello Oza, I do recall our discussions. Since both you and Vinayak use non-work email addresses, I thought maybe you both worked for the same company, and he had taken over the task. ;-) I do like the simplicity of Vinayak's latest patch-set, but it would be ideal if it could be restricted to 64-bit and just use the original RAM dump file. It's also reasonable to offer an option to save it as separate ELF file. So I don't think it would take much to update his patch to get that done. But certainly you can post your version, or perhaps patch Vinayak's, as long as the end result covers the requests above. Thanks, Dave

Hello Vinayak, This patch works and tests well, although I wish I had a more useful ARM64 sample dumpfile whose kernel has come up farther than the last one, and that either has kernel modules or at least one user-space task so I could verify the VTOP of those types of virtual addresses. Here are a few minor issues with ramdump.c: (1) It #includes <libelf.h>, which would require a new package to be installed: $ rpm -qf /usr/include/libelf.h elfutils-libelf-devel $ I changed it to #include <elf.h> (from glibc-headers) and it compiles just fine. (2) The ramdump_def structure: struct ramdump_def { char *path; int rfd; ulong start_paddr; ulong end_paddr; }; The start_paddr and end_paddr fields have to be 64-bits in length to account for ARM kernels that have RAM beyond 4GB. They start_paddr member gets assigned to a 64-bit p_paddr field and the end_paddr field would overflow in a greater-than-4GB scenario. They should be ulonglong or physaddr_t types. Related to that, the start_paddr field should be intitialized with htoll() instead of htol() just in case a 32-bit node starts above the 32-bit threshold. (3) The "Skipping dumpfile size check for ramdump" is unnecessary given that the end_paddr is calculated based upon the size of the RAM dump file itself. (4) The "converting ramdump to kdump" message is unnecessary, and the "Creating kdump ELF <filename>" should be different if only an ELF header is created vs. whether a completely new ELF file is being created. I've made changes above to your patch with no problems, so there's no need for a re-post. I'll keep playing around to see if anything else comes up. Nice job! Thanks, Dave ----- Original Message -----