> original non-g built vmlinux, things would work. > > In your case, it's defaulting to the /usr/src/linux/System.map, which is > associated with the vmlinux that you built with -g. > > Presuming you have the /boot/System.map associated with the original > distro's vmlinux file (the kernel that's running), what happens if you do > this: > > $ crash vmlinux /boot/System.map [vmcore]

Badari Pulavarty wrote: >... > > > >>>original non-g built vmlinux, things would work. >>> >>>In your case, it's defaulting to the /usr/src/linux/System.map, which is >>>associated with the vmlinux that you built with -g. >>> >>>Presuming you have the /boot/System.map associated with the original >>>distro's vmlinux file (the kernel that's running), what happens if you do >>>this: >>> >>>$ crash vmlinux /boot/System.map [vmcore] >>> >>> >Much better with /boot/System.map. > > > Yay! So, we seem to have resolved the funky text disassembly issue... But I still would like to continue working with the vmcore -- I'm sitting here with Vivek, and he will grab a copy of the System.map. Thanks, (and phew...) Dave >elm3a242:/usr/src/linux # /root/cr*23/crash vmlinux /boot/System.map-2.6.16-20-smp > >crash 4.0-2.23 >Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc. >Copyright (C) 2004, 2005, 2006 IBM Corporation >Copyright (C) 1999-2006 Hewlett-Packard Co >Copyright (C) 2005 Fujitsu Limited >Copyright (C) 2005 NEC Corporation >Copyright (C) 1999, 2002 Silicon Graphics, Inc. >Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. >This program is free software, covered by the GNU General Public >License, >and you are welcome to change it and/or distribute copies of it under >certain conditions. Enter "help copying" to see the conditions. >This program has absolutely no warranty. Enter "help warranty" for >details. > >GNU gdb 6.1 >Copyright 2004 Free Software Foundation, Inc. >GDB is free software, covered by the GNU General Public License, and you >are >welcome to change it and/or distribute copies of it under certain >conditions. >Type "show copying" to see the conditions. >There is absolutely no warranty for GDB. Type "show warranty" for >details. >This GDB was configured as "x86_64-unknown-linux-gnu"... > > SYSTEM MAP: /boot/System.map-2.6.16-20-smp >DEBUG KERNEL: vmlinux (2.6.16-20-smp) > DUMPFILE: /dev/mem > CPUS: 2 > DATE: Thu Apr 27 03:18:57 2006 > UPTIME: 16:47:17 >LOAD AVERAGE: 0.00, 0.00, 0.00 > TASKS: 64 > NODENAME: elm3a242 > RELEASE: 2.6.16-20-smp > VERSION: #1 SMP Mon Apr 10 04:51:13 UTC 2006 > MACHINE: x86_64 (3000 Mhz) > MEMORY: 4.6 GB > PID: 12963 > COMMAND: "crash" > TASK: ffff810121ef5790 [THREAD_INFO: ffff81011d3c4000] > CPU: 1 > STATE: TASK_RUNNING (ACTIVE) > >crash> dis sys_read >0xffffffff8017b751 <sys_read>: push %r13 >0xffffffff8017b753 <sys_read+2>: mov %rsi,%r13 >0xffffffff8017b756 <sys_read+5>: push %r12 >0xffffffff8017b758 <sys_read+7>: mov $0xfffffffffffffff7,%r12 >0xffffffff8017b75f <sys_read+14>: push %rbp >0xffffffff8017b760 <sys_read+15>: mov %rdx,%rbp >0xffffffff8017b763 <sys_read+18>: push %rbx >0xffffffff8017b764 <sys_read+19>: sub $0x18,%rsp >0xffffffff8017b768 <sys_read+23>: lea 0x14(%rsp),%rsi >0xffffffff8017b76d <sys_read+28>: callq 0xffffffff8017bce3 ><fget_light> >0xffffffff8017b772 <sys_read+33>: test %rax,%rax >0xffffffff8017b775 <sys_read+36>: mov %rax,%rbx >0xffffffff8017b778 <sys_read+39>: > je 0xffffffff8017b7b1 <sys_read+96> >0xffffffff8017b77a <sys_read+41>: mov 0x38(%rax),%rax >0xffffffff8017b77e <sys_read+45>: lea 0x8(%rsp),%rcx >0xffffffff8017b783 <sys_read+50>: mov %rbp,%rdx >0xffffffff8017b786 <sys_read+53>: mov %r13,%rsi >0xffffffff8017b789 <sys_read+56>: mov %rbx,%rdi >0xffffffff8017b78c <sys_read+59>: mov %rax,0x8(%rsp) >0xffffffff8017b791 <sys_read+64>: callq 0xffffffff8017b2ef ><vfs_read> >0xffffffff8017b796 <sys_read+69>: mov %rax,%r12 >0xffffffff8017b799 <sys_read+72>: mov 0x8(%rsp),%rax > >-- >Crash-utility mailing list &gt;Crash-utility(a)redhat.com >https://www.redhat.com/mailman/listinfo/crash-utility > > -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

David Wilder wrote: > Dave Anderson wrote: > > >Badari Pulavarty wrote: > > > > > > > >>... > >> > >> > >> > >>>>original non-g built vmlinux, things would work. > >>>> > >>>>In your case, it's defaulting to the /usr/src/linux/System.map, > which is > >>>>associated with the vmlinux that you built with -g. > >>>> > >>>>Presuming you have the /boot/System.map associated with the original > >>>>distro's vmlinux file (the kernel that's running), what happens > if you do > >>>>this: > >>>> > >>>>$ crash vmlinux /boot/System.map [vmcore] > >>>> > >>>> > >>Much better with /boot/System.map. > >> > >> > >> > > > >Yay! > > > >So, we seem to have resolved the funky text disassembly issue... > > > >But I still would like to continue working with the vmcore -- I'm > sitting here > >with Vivek, and he will grab a copy of the System.map. > > > >Thanks, (and phew...) > > > >Dave > > > > > > > > > >>elm3a242:/usr/src/linux # /root/cr*23/crash vmlinux > /boot/System.map-2.6.16-20-smp > >> > >>crash 4.0-2.23 > >>Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc. > >>Copyright (C) 2004, 2005, 2006 IBM Corporation > >>Copyright (C) 1999-2006 Hewlett-Packard Co > >>Copyright (C) 2005 Fujitsu Limited > >>Copyright (C) 2005 NEC Corporation > >>Copyright (C) 1999, 2002 Silicon Graphics, Inc. > >>Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > >>This program is free software, covered by the GNU General Public > >>License, > >>and you are welcome to change it and/or distribute copies of it under > >>certain conditions. Enter "help copying" to see the conditions. > >>This program has absolutely no warranty. Enter "help warranty" for > >>details. > >> > >>GNU gdb 6.1 > >>Copyright 2004 Free Software Foundation, Inc. > >>GDB is free software, covered by the GNU General Public License, > and you > >>are > >>welcome to change it and/or distribute copies of it under certain > >>conditions. > >>Type "show copying" to see the conditions. > >>There is absolutely no warranty for GDB. Type "show warranty" for > >>details. > >>This GDB was configured as "x86_64-unknown-linux-gnu"... > >> > >> SYSTEM MAP: /boot/System.map-2.6.16-20-smp > >>DEBUG KERNEL: vmlinux (2.6.16-20-smp) > >> DUMPFILE: /dev/mem > >> CPUS: 2 > >> DATE: Thu Apr 27 03:18:57 2006 > >> UPTIME: 16:47:17 > >>LOAD AVERAGE: 0.00, 0.00, 0.00 > >> TASKS: 64 > >> NODENAME: elm3a242 > >> RELEASE: 2.6.16-20-smp > >> VERSION: #1 SMP Mon Apr 10 04:51:13 UTC 2006 > >> MACHINE: x86_64 (3000 Mhz) > >> MEMORY: 4.6 GB > >> PID: 12963 > >> COMMAND: "crash" > >> TASK: ffff810121ef5790 [THREAD_INFO: ffff81011d3c4000] > >> CPU: 1 > >> STATE: TASK_RUNNING (ACTIVE) > >> > >>crash> dis sys_read > >>0xffffffff8017b751 <sys_read>: push %r13 > >>0xffffffff8017b753 <sys_read+2>: mov %rsi,%r13 > >>0xffffffff8017b756 <sys_read+5>: push %r12 > >>0xffffffff8017b758 <sys_read+7>: mov > $0xfffffffffffffff7,%r12 > >>0xffffffff8017b75f <sys_read+14>: push %rbp > >>0xffffffff8017b760 <sys_read+15>: mov %rdx,%rbp > >>0xffffffff8017b763 <sys_read+18>: push %rbx > >>0xffffffff8017b764 <sys_read+19>: sub $0x18,%rsp > >>0xffffffff8017b768 <sys_read+23>: lea 0x14(%rsp),%rsi > >>0xffffffff8017b76d <sys_read+28>: callq 0xffffffff8017bce3 > >><fget_light> > >>0xffffffff8017b772 <sys_read+33>: test %rax,%rax > >>0xffffffff8017b775 <sys_read+36>: mov %rax,%rbx > >>0xffffffff8017b778 <sys_read+39>: > >> je 0xffffffff8017b7b1 <sys_read+96> > >>0xffffffff8017b77a <sys_read+41>: mov 0x38(%rax),%rax > >>0xffffffff8017b77e <sys_read+45>: lea 0x8(%rsp),%rcx > >>0xffffffff8017b783 <sys_read+50>: mov %rbp,%rdx > >>0xffffffff8017b786 <sys_read+53>: mov %r13,%rsi > >>0xffffffff8017b789 <sys_read+56>: mov %rbx,%rdi > >>0xffffffff8017b78c <sys_read+59>: mov %rax,0x8(%rsp) > >>0xffffffff8017b791 <sys_read+64>: callq 0xffffffff8017b2ef > >><vfs_read> > >>0xffffffff8017b796 <sys_read+69>: mov %rax,%r12 > >>0xffffffff8017b799 <sys_read+72>: mov 0x8(%rsp),%rax > >> > >>-- > >>Crash-utility mailing list > &gt;&gt;Crash-utility(a)redhat.com > >>https://www.redhat.com/mailman/listinfo/crash-utility > >> > >> > > > >-- > >Crash-utility mailing list > &gt;Crash-utility(a)redhat.com > >https://www.redhat.com/mailman/listinfo/crash-utility > > > > > > > Dave I am seeing a similar problem to Badari, however in my case I am > running the kernel built with -g. > I can run crash live fine. But running on a vmcore I get the following > error. > > deimos:/var/log/dump/2006-05-02-13:10 # ~/crash-4.0-2.24/crash > /usr/src/linux/vmlinux ./vmcore > > crash 4.0-2.24 > Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006 IBM Corporation > Copyright (C) 1999-2006 Hewlett-Packard Co > Copyright (C) 2005 Fujitsu Limited > Copyright (C) 2005 NEC Corporation > Copyright (C) 1999, 2002 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public > License, > and you are welcome to change it and/or distribute copies of it under > certain conditions. Enter "help copying" to see the conditions. > This program has absolutely no warranty. Enter "help warranty" for > details. > > GNU gdb 6.1 > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and > you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for > details. > This GDB was configured as "powerpc64-unknown-linux-gnu"... > > Wilder vmlist = 0xc00000000fa26940 > crash: read error: kernel virtual address: c00000000fa26940 type: "first > vmlist > addr" > > Errors like the one above typically occur when the kernel and memory > source > do not match. These are the files being used: > > KERNEL: /usr/src/linux/vmlinux > DUMPFILE: ./vmcore > > -- > When I check System.map for vmlist or do a "nm vmlinux | grep vmlist" I > get a different number than shown above. > I tried specifying the System.map file on the command line but no > difference. > > Hi Dave, Like Badari's issue, which actually was a non-issue, it's difficult to debug from here. A couple things... When you use an additional System.map argument on the command line, you should see the message stating that several thousand symbol values are being changed from what is in the vmlinux file to what is in the System.map file: please wait... (patching ##### gdb minimal_symbol values) This gets kicked off in the gdb_interface.c gdb_session_init() function here: /* * Patch gdb's symbol values with the correct values from either * the System.map or non-debug vmlinux, whichever is in effect. */ if ((pc->flags & SYSMAP) || (pc->namelist_debug && !pc->debuginfo_file)) { req->command = GNU_PATCH_SYMBOL_VALUES; req->flags = GNU_RETURN_ON_ERROR; gdb_interface(req); if (req->flags & GNU_COMMAND_FAILED) error(FATAL, "patching of gdb symbol values failed\n"); } So, presuming that worked OK, what's strange is the read error you get on the vmcore: >crash: read error: kernel virtual address: c00000000fa26940 type: "first vmlist addr" > The first_vmalloc_address() function is simply reading reading the contents of the vmlist pointer, which points to the first vm_struct in the list, and which is returning c00000000fa26940. But then it fails to read what's in the "addr" offset of the vm_struct at that address: get_symbol_data("vmlist", sizeof(void *), &vmlist); if (!readmem(vmlist+OFFSET(vm_struct_addr), KVADDR, &addr, sizeof(void *), "first vmlist addr", RETURN_ON_ERROR)) non_matching_kernel(); Since the "addr" offset is 0, it just reads address c00000000fa26940, which is a just a kmalloc'd unity-mapped address. So the question is, why is that address not capable of being read from the vmcore? If you do a "help -n" to dump the header contents of the vmcore, does physical address fa26940 not fit into any of the PT_LOAD segments? (Again -- this all presumes the the "vmlist" symbol address is correct...) Dave ------------------------------------------------------------------------ -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

Hi Dave, Like Badari's issue, which actually was a non-issue, it's difficult to debug from here. A couple things... When you use an additional System.map argument on the command line, you should see the message stating that several thousand symbol values are being changed from what is in the vmlinux file to what is in the System.map file: please wait... (patching ##### gdb minimal_symbol values) This gets kicked off in the gdb_interface.c gdb_session_init() function here: /* * Patch gdb's symbol values with the correct values from either * the System.map or non-debug vmlinux, whichever is in effect. */ if ((pc->flags & SYSMAP) || (pc->namelist_debug && !pc->debuginfo_file)) { req->command = GNU_PATCH_SYMBOL_VALUES; req->flags = GNU_RETURN_ON_ERROR; gdb_interface(req); if (req->flags & GNU_COMMAND_FAILED) error(FATAL, "patching of gdb symbol values failed\n"); } So, presuming that worked OK, what's strange is the read error you get on the vmcore: >crash: read error: kernel virtual address: c00000000fa26940 type: "first vmlist addr" > The first_vmalloc_address() function is simply reading reading the contents of the vmlist pointer, which points to the first vm_struct in the list, and which is returning c00000000fa26940. But then it fails to read what's in the "addr" offset of the vm_struct at that address: get_symbol_data("vmlist", sizeof(void *), &vmlist); if (!readmem(vmlist+OFFSET(vm_struct_addr), KVADDR, &addr, sizeof(void *), "first vmlist addr", RETURN_ON_ERROR)) non_matching_kernel(); Since the "addr" offset is 0, it just reads address c00000000fa26940, which is a just a kmalloc'd unity-mapped address. So the question is, why is that address not capable of being read from the vmcore? If you do a "help -n" to dump the header contents of the vmcore, does physical address fa26940 not fit into any of the PT_LOAD segments? (Again -- this all presumes the the "vmlist" symbol address is correct...) Dave ------------------------------------------------------------------------ -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

David Wilder wrote: > > crash -d 2 gave me the header.. > Elf64_Phdr: > p_type: 4 (PT_NOTE) > p_offset: 288 (120) > p_vaddr: 0 > p_paddr: 0 > p_filesz: 1048 (418) > p_memsz: 1048 (418) > p_flags: 0 () > p_align: 0 > Elf64_Phdr: > p_type: 1 (PT_LOAD) > p_offset: 1336 (538) > p_vaddr: c000000000000000 > p_paddr: 0 > p_filesz: 32768 (8000) > p_memsz: 32768 (8000) > p_flags: 7 (PF_X|PF_W|PF_R) > p_align: 0 > Elf64_Phdr: > p_type: 1 (PT_LOAD) > p_offset: 34104 (8538) > p_vaddr: c000000000008000 > p_paddr: 8000 > p_filesz: 33521663 (1ff7fff) > p_memsz: 33521663 (1ff7fff) > p_flags: 7 (PF_X|PF_W|PF_R) > p_align: 0 > Elf64_Phdr: > p_type: 1 (PT_LOAD) > p_offset: 33555767 (2000537) > p_vaddr: c00000002fd0f001 > p_paddr: 2fd0f001 > p_filesz: 3492745215 (d02f0fff) > p_memsz: 3492745215 (d02f0fff) > p_flags: 7 (PF_X|PF_W|PF_R) > p_align: 0 > > Looks like that address is in a hole between PT_LOAD segments 2 and 3. Right, but something is fundamentally wrong with your vmcore: p_paddr p_filesz First segment: 0 8000 Second segment: 8000 1ff7fff Third segment: 2fd0f001 d02f0fff It cannot have a p_paddr address that is not page-aligned, nor can you have a p_filesz that is not page-aligned. Dave

On Thu, May 04, 2006 at 03:08:14PM -0400, Dave Anderson wrote: > > > > Right, but something is fundamentally wrong with your > > vmcore: > > > > p_paddr p_filesz > > First segment: 0 8000 > > Second segment: 8000 1ff7fff > > Third segment: 2fd0f001 d02f0fff > > > > It cannot have a p_paddr address that is not page-aligned, > > nor can you have a p_filesz that is not page-aligned. > > > > Dave > > Vivek et al, > > Have there been any recent changes to the kdump code > that might cause strange PT_LOAD segment contents? > Dave, As per my knowledge nothing has changed. Generally PT_LOAD headers are generated from user space while kernel is being loaded and second kernel leaves them untouched except updating the offset field. So there are high chances that user space itself is not generating right headers while loading the capture kernel. Dave Wilder, can you put some debugging statement in kexec-tools and see what header values are being generated from user space and do they match with what /proc/vmcore is showing. I will do some experiments on x86_64. Thanks Vivek