8:55 p.m.
I have a dump from RHEL 6.2 with all these masks set to all f's, so obviously crash
excludes them.
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"possible"
WARNING: cannot read cpu_possible_map
crash: page excluded: kernel virtual address: ffffffffffffffff type: "present"
WARNING: cannot read cpu_present_map
crash: page excluded: kernel virtual address: ffffffffffffffff type: "online"
WARNING: cannot read cpu_online_map
typedef struct cpumask { DECLARE_BITMAP(bits, NR_CPUS); } cpumask_t;
#ifndef CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS
/* These strip const, as traditionally they weren't const. */
#define cpu_possible_map (*(cpumask_t *)cpu_possible_mask)
#define cpu_online_map (*(cpumask_t *)cpu_online_mask)
#define cpu_present_map (*(cpumask_t *)cpu_present_mask)
#define cpu_active_map (*(cpumask_t *)cpu_active_mask)
Are these deprecated now and is that why they are fixed at all f's.
[root@centos62 crash]# crash System.map vmlinux va102.2012-06-14.vmcore
crash 6.0.7
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"possible"
WARNING: cannot read cpu_possible_map
crash: page excluded: kernel virtual address: ffffffffffffffff type: "present"
WARNING: cannot read cpu_present_map
crash: page excluded: kernel virtual address: ffffffffffffffff type: "online"
WARNING: cannot read cpu_online_map
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"cpu_present_map"
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"cpu_present_map"
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"cpu_online_map"
SYSTEM MAP: System.map
DEBUG KERNEL: vmlinux (2.6.32-220.7.1.el6)
DUMPFILE: va102.2012-06-14.vmcore [PARTIAL DUMP]
CPUS: 32
DATE: Thu Jun 14 05:24:17 2012
UPTIME: 11:13:23
LOAD AVERAGE: 215.00, 196.78, 135.81
TASKS: 1778
NODENAME: va1splunkindex02.va1.swtor.int
RELEASE: 2.6.32-220.7.1.el6.x86_64
VERSION: #1 SMP Tue Mar 6 13:54:51 EST 2012
MACHINE: x86_64 (2799 Mhz)
MEMORY: 48 GB
PANIC: "Oops: 0002 [#1] SMP " (check log for details)
PID: 2428
COMMAND: "bash"
TASK: ffff88063874ab40 [THREAD_INFO: ffff8806dcd1a000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
Thanks
Laurence
8:21 a.m.
New subject: Pages excluded in dump RHEL 6.2 - set to ffffffffffffffff
----- Original Message -----
I have a dump from RHEL 6.2 with all these masks set to all f's,
so
obviously crash excludes them.
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"possible"
WARNING: cannot read cpu_possible_map
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"present"
WARNING: cannot read cpu_present_map
crash: page excluded: kernel virtual address: ffffffffffffffff type: "online"
WARNING: cannot read cpu_online_map
typedef struct cpumask { DECLARE_BITMAP(bits, NR_CPUS); } cpumask_t;
#ifndef CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS
/* These strip const, as traditionally they weren't const. */
#define cpu_possible_map (*(cpumask_t *)cpu_possible_mask)
#define cpu_online_map (*(cpumask_t *)cpu_online_mask)
#define cpu_present_map (*(cpumask_t *)cpu_present_mask)
#define cpu_active_map (*(cpumask_t *)cpu_active_mask)
Are these deprecated now and is that why they are fixed at all f's.
No, that's just a common error message for accessing the maps or masks.
When reading them, crash checks first whether there is the older kernel symbol
named "cpu_xxx_map", and if it doesn't exists, then it checks for the
symbol
named "cpu_xxx_mask". It then tried to read the memory that those symbols
point to, which all contained values of ffffffffffffffff. So clearly
there's some kind of mis-match between the symbol values -- modified by
using a System.map file -- and what's in the vmcore file.
(FWIW, the dumpfile couldn't find the page for bogus address, and presumed
that it was "excluded" since it's a filtered dump, so that's a
non-issue...)
First question: it appears that you are analyzing a RHEL6 kernel, so why are
you using a "System.map" argument? Did you rebuild the kernel but when you
did that, you didn't save the rebuilt vmlinux file?
Dave
[root@centos62 crash]# crash System.map vmlinux va102.2012-06-14.vmcore
crash 6.0.7
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"possible"
WARNING: cannot read cpu_possible_map
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"present"
WARNING: cannot read cpu_present_map
crash: page excluded: kernel virtual address: ffffffffffffffff type: "online"
WARNING: cannot read cpu_online_map
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"cpu_present_map"
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"cpu_present_map"
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"cpu_online_map"
SYSTEM MAP: System.map
DEBUG KERNEL: vmlinux (2.6.32-220.7.1.el6)
DUMPFILE: va102.2012-06-14.vmcore [PARTIAL DUMP]
CPUS: 32
DATE: Thu Jun 14 05:24:17 2012
UPTIME: 11:13:23
LOAD AVERAGE: 215.00, 196.78, 135.81
TASKS: 1778
NODENAME: va1splunkindex02.va1.swtor.int
RELEASE: 2.6.32-220.7.1.el6.x86_64
VERSION: #1 SMP Tue Mar 6 13:54:51 EST 2012
MACHINE: x86_64 (2799 Mhz)
MEMORY: 48 GB
PANIC: "Oops: 0002 [#1] SMP " (check log for details)
PID: 2428
COMMAND: "bash"
TASK: ffff88063874ab40 [THREAD_INFO: ffff8806dcd1a000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
Thanks
Laurence
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
9:25 a.m.
New subject: Pages excluded in dump RHEL 6.2 - set to ffffffffffffffff
Hi Dave,
OK, makes sense. I still spend 95% of my time in the RHEL 5 kernels so it's just lack
of enough time in the RHEL 6 kernels.
I did not have the debuginfo so I built a vmkernel with debug from source and used that. I
was able to get what I wanted out of the dump, specifically needed to look at the net_dev
structures.
Is there a write up somewhere on changes with crash for the 2.6.32.x kernels that you can
share.
Thanks
Laurence
-----Original Message-----
From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf
Of Dave Anderson
Sent: Wednesday, June 20, 2012 9:22 AM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] Pages excluded in dump RHEL 6.2 - set to ffffffffffffffff
----- Original Message -----
I have a dump from RHEL 6.2 with all these masks set to all f's,
so
obviously crash excludes them.
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"possible"
WARNING: cannot read cpu_possible_map
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"present"
WARNING: cannot read cpu_present_map
crash: page excluded: kernel virtual address: ffffffffffffffff type: "online"
WARNING: cannot read cpu_online_map
typedef struct cpumask { DECLARE_BITMAP(bits, NR_CPUS); } cpumask_t;
#ifndef CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS
/* These strip const, as traditionally they weren't const. */
#define cpu_possible_map (*(cpumask_t *)cpu_possible_mask)
#define cpu_online_map (*(cpumask_t *)cpu_online_mask)
#define cpu_present_map (*(cpumask_t *)cpu_present_mask)
#define cpu_active_map (*(cpumask_t *)cpu_active_mask)
Are these deprecated now and is that why they are fixed at all f's.
No, that's just a common error message for accessing the maps or masks.
When reading them, crash checks first whether there is the older kernel symbol named
"cpu_xxx_map", and if it doesn't exists, then it checks for the symbol named
"cpu_xxx_mask". It then tried to read the memory that those symbols point to,
which all contained values of ffffffffffffffff. So clearly there's some kind of
mis-match between the symbol values -- modified by using a System.map file -- and
what's in the vmcore file.
(FWIW, the dumpfile couldn't find the page for bogus address, and presumed that it was
"excluded" since it's a filtered dump, so that's a non-issue...)
First question: it appears that you are analyzing a RHEL6 kernel, so why are you using a
"System.map" argument? Did you rebuild the kernel but when you did that, you
didn't save the rebuilt vmlinux file?
Dave
[root@centos62 crash]# crash System.map vmlinux
va102.2012-06-14.vmcore
crash 6.0.7
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C)
1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012
Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002,
2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License, and you are welcome to change it and/or distribute copies of
it under certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"possible"
WARNING: cannot read cpu_possible_map
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"present"
WARNING: cannot read cpu_present_map
crash: page excluded: kernel virtual address: ffffffffffffffff type: "online"
WARNING: cannot read cpu_online_map
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"cpu_present_map"
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"cpu_present_map"
crash: page excluded: kernel virtual address: ffffffffffffffff type:
"cpu_online_map"
SYSTEM MAP: System.map
DEBUG KERNEL: vmlinux (2.6.32-220.7.1.el6)
DUMPFILE: va102.2012-06-14.vmcore [PARTIAL DUMP]
CPUS: 32
DATE: Thu Jun 14 05:24:17 2012
UPTIME: 11:13:23
LOAD AVERAGE: 215.00, 196.78, 135.81
TASKS: 1778
NODENAME: va1splunkindex02.va1.swtor.int
RELEASE: 2.6.32-220.7.1.el6.x86_64
VERSION: #1 SMP Tue Mar 6 13:54:51 EST 2012
MACHINE: x86_64 (2799 Mhz)
MEMORY: 48 GB
PANIC: "Oops: 0002 [#1] SMP " (check log for details)
PID: 2428
COMMAND: "bash"
TASK: ffff88063874ab40 [THREAD_INFO: ffff8806dcd1a000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
Thanks
Laurence
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
9:42 a.m.
New subject: Pages excluded in dump RHEL 6.2 - set to ffffffffffffffff
----- Original Message -----
Hi Dave,
OK, makes sense. I still spend 95% of my time in the RHEL 5 kernels
so it's just lack of enough time in the RHEL 6 kernels.
I did not have the debuginfo so I built a vmkernel with debug from
source and used that. I was able to get what I wanted out of the
dump, specifically needed to look at the net_dev structures.
Is there a write up somewhere on changes with crash for the 2.6.32.x
kernels that you can share.
No, not specifically for the 2.6.32 or any other kernel version.
The crash utility is constantly updated to handle the shifting sands
of the underlying Linux kernel. But it's also kept backwards-compatible,
so the crash sources are full of checks for kernel differences such as
the cpu maps to mask changes.
But I'm still not clear why you are reading the ffffffffffffffff's at
those symbolic locations if the System.map's symbol values (which
over-write the ones in the vmlinux file) match what's in the dumped
kernel? I believe that the vmlinux file that you built -- even with
the correct System.map -- is different enough from the crashed kernel
so that there are inequities that cannot be overcome with just a System.map.
Dave
Thanks
Laurence
-----Original Message-----
From: crash-utility-bounces(a)redhat.com
[mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson
Sent: Wednesday, June 20, 2012 9:22 AM
To: Discussion list for crash utility usage, maintenance and
development
Subject: Re: [Crash-utility] Pages excluded in dump RHEL 6.2 - set to
ffffffffffffffff
----- Original Message -----
> I have a dump from RHEL 6.2 with all these masks set to all f's, so
> obviously crash excludes them.
>
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "possible"
> WARNING: cannot read cpu_possible_map
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "present"
> WARNING: cannot read cpu_present_map
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "online"
> WARNING: cannot read cpu_online_map
>
> typedef struct cpumask { DECLARE_BITMAP(bits, NR_CPUS); }
> cpumask_t;
>
> #ifndef CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS
> /* These strip const, as traditionally they weren't const. */
> #define cpu_possible_map (*(cpumask_t *)cpu_possible_mask)
> #define cpu_online_map (*(cpumask_t *)cpu_online_mask)
> #define cpu_present_map (*(cpumask_t *)cpu_present_mask)
> #define cpu_active_map (*(cpumask_t *)cpu_active_mask)
>
> Are these deprecated now and is that why they are fixed at all f's.
No, that's just a common error message for accessing the maps or
masks.
When reading them, crash checks first whether there is the older
kernel symbol named "cpu_xxx_map", and if it doesn't exists, then it
checks for the symbol named "cpu_xxx_mask". It then tried to read
the memory that those symbols point to, which all contained values
of ffffffffffffffff. So clearly there's some kind of mis-match
between the symbol values -- modified by using a System.map file --
and what's in the vmcore file.
(FWIW, the dumpfile couldn't find the page for bogus address, and
presumed that it was "excluded" since it's a filtered dump, so
that's a non-issue...)
First question: it appears that you are analyzing a RHEL6 kernel, so
why are you using a "System.map" argument? Did you rebuild the
kernel but when you did that, you didn't save the rebuilt vmlinux
file?
Dave
>
> [root@centos62 crash]# crash System.map vmlinux
> va102.2012-06-14.vmcore
>
> crash 6.0.7
> Copyright (C) 2002-2012 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C)
> 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012
> Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan
> K.K.
> Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002,
> 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License, and you are welcome to change it and/or distribute copies
> of
> it under certain conditions. Enter "help copying" to see the
> conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "possible"
> WARNING: cannot read cpu_possible_map
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "present"
> WARNING: cannot read cpu_present_map
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "online"
> WARNING: cannot read cpu_online_map
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "cpu_present_map"
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "cpu_present_map"
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "cpu_online_map"
> SYSTEM MAP: System.map
> DEBUG KERNEL: vmlinux (2.6.32-220.7.1.el6)
> DUMPFILE: va102.2012-06-14.vmcore [PARTIAL DUMP]
> CPUS: 32
> DATE: Thu Jun 14 05:24:17 2012
> UPTIME: 11:13:23
> LOAD AVERAGE: 215.00, 196.78, 135.81
> TASKS: 1778
> NODENAME: va1splunkindex02.va1.swtor.int
> RELEASE: 2.6.32-220.7.1.el6.x86_64
> VERSION: #1 SMP Tue Mar 6 13:54:51 EST 2012
> MACHINE: x86_64 (2799 Mhz)
> MEMORY: 48 GB
> PANIC: "Oops: 0002 [#1] SMP " (check log for details)
> PID: 2428
> COMMAND: "bash"
> TASK: ffff88063874ab40 [THREAD_INFO: ffff8806dcd1a000]
> CPU: 0
> STATE: TASK_RUNNING (PANIC)
>
> Thanks
> Laurence
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
10:18 a.m.
New subject: Pages excluded in dump RHEL 6.2 - set to ffffffffffffffff
Dave,
You had it exactly right:
Using the generic System.map-2.6.32-220.7.1.el6.x86_64 works fine. The System.map from my
build was missing symbols.
Thanks!
Laurence
crash 6.0.7
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
SYSTEM MAP: System.map-2.6.32-220.7.1.el6.x86_64
DEBUG KERNEL: vmlinux (2.6.32-220.7.1.el6)
DUMPFILE: va102.2012-06-14.vmcore [PARTIAL DUMP]
CPUS: 24
DATE: Thu Jun 14 05:24:17 2012
UPTIME: 11:13:23
LOAD AVERAGE: 215.00, 196.78, 135.81
TASKS: 1778
NODENAME: va1splunkindex02.va1.swtor.int
RELEASE: 2.6.32-220.7.1.el6.x86_64
VERSION: #1 SMP Tue Mar 6 13:54:51 EST 2012
MACHINE: x86_64 (2799 Mhz)
MEMORY: 48 GB
PANIC: "Oops: 0002 [#1] SMP " (check log for details)
PID: 2428
COMMAND: "bash"
TASK: ffff88063874ab40 [THREAD_INFO: ffff8806dcd1a000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
-----Original Message-----
From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf
Of Dave Anderson
Sent: Wednesday, June 20, 2012 10:42 AM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] Pages excluded in dump RHEL 6.2 - set to ffffffffffffffff
----- Original Message -----
Hi Dave,
OK, makes sense. I still spend 95% of my time in the RHEL 5 kernels so
it's just lack of enough time in the RHEL 6 kernels.
I did not have the debuginfo so I built a vmkernel with debug from
source and used that. I was able to get what I wanted out of the dump,
specifically needed to look at the net_dev structures.
Is there a write up somewhere on changes with crash for the 2.6.32.x
kernels that you can share.
No, not specifically for the 2.6.32 or any other kernel version.
The crash utility is constantly updated to handle the shifting sands of the underlying
Linux kernel. But it's also kept backwards-compatible, so the crash sources are full
of checks for kernel differences such as the cpu maps to mask changes.
But I'm still not clear why you are reading the ffffffffffffffff's at those
symbolic locations if the System.map's symbol values (which over-write the ones in the
vmlinux file) match what's in the dumped kernel? I believe that the vmlinux file that
you built -- even with the correct System.map -- is different enough from the crashed
kernel so that there are inequities that cannot be overcome with just a System.map.
Dave
Thanks
Laurence
-----Original Message-----
From: crash-utility-bounces(a)redhat.com
[mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson
Sent: Wednesday, June 20, 2012 9:22 AM
To: Discussion list for crash utility usage, maintenance and
development
Subject: Re: [Crash-utility] Pages excluded in dump RHEL 6.2 - set to
ffffffffffffffff
----- Original Message -----
> I have a dump from RHEL 6.2 with all these masks set to all f's, so
> obviously crash excludes them.
>
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "possible"
> WARNING: cannot read cpu_possible_map
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "present"
> WARNING: cannot read cpu_present_map
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "online"
> WARNING: cannot read cpu_online_map
>
> typedef struct cpumask { DECLARE_BITMAP(bits, NR_CPUS); } cpumask_t;
>
> #ifndef CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS
> /* These strip const, as traditionally they weren't const. */
> #define cpu_possible_map (*(cpumask_t *)cpu_possible_mask)
> #define cpu_online_map (*(cpumask_t *)cpu_online_mask)
> #define cpu_present_map (*(cpumask_t *)cpu_present_mask)
> #define cpu_active_map (*(cpumask_t *)cpu_active_mask)
>
> Are these deprecated now and is that why they are fixed at all f's.
No, that's just a common error message for accessing the maps or
masks.
When reading them, crash checks first whether there is the older
kernel symbol named "cpu_xxx_map", and if it doesn't exists, then it
checks for the symbol named "cpu_xxx_mask". It then tried to read the
memory that those symbols point to, which all contained values of
ffffffffffffffff. So clearly there's some kind of mis-match between
the symbol values -- modified by using a System.map file -- and what's
in the vmcore file.
(FWIW, the dumpfile couldn't find the page for bogus address, and
presumed that it was "excluded" since it's a filtered dump, so that's
a non-issue...)
First question: it appears that you are analyzing a RHEL6 kernel, so
why are you using a "System.map" argument? Did you rebuild the kernel
but when you did that, you didn't save the rebuilt vmlinux file?
Dave
>
> [root@centos62 crash]# crash System.map vmlinux
> va102.2012-06-14.vmcore
>
> crash 6.0.7
> Copyright (C) 2002-2012 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C)
> 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012
> Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan
> K.K.
> Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002,
> 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License, and you are welcome to change it and/or distribute copies
> of it under certain conditions. Enter "help copying" to see the
> conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "possible"
> WARNING: cannot read cpu_possible_map
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "present"
> WARNING: cannot read cpu_present_map
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "online"
> WARNING: cannot read cpu_online_map
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "cpu_present_map"
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "cpu_present_map"
> crash: page excluded: kernel virtual address: ffffffffffffffff
> type: "cpu_online_map"
> SYSTEM MAP: System.map
> DEBUG KERNEL: vmlinux (2.6.32-220.7.1.el6)
> DUMPFILE: va102.2012-06-14.vmcore [PARTIAL DUMP]
> CPUS: 32
> DATE: Thu Jun 14 05:24:17 2012
> UPTIME: 11:13:23
> LOAD AVERAGE: 215.00, 196.78, 135.81
> TASKS: 1778
> NODENAME: va1splunkindex02.va1.swtor.int
> RELEASE: 2.6.32-220.7.1.el6.x86_64
> VERSION: #1 SMP Tue Mar 6 13:54:51 EST 2012
> MACHINE: x86_64 (2799 Mhz)
> MEMORY: 48 GB
> PANIC: "Oops: 0002 [#1] SMP " (check log for details)
> PID: 2428
> COMMAND: "bash"
> TASK: ffff88063874ab40 [THREAD_INFO: ffff8806dcd1a000]
> CPU: 0
> STATE: TASK_RUNNING (PANIC)
>
> Thanks
> Laurence
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
4396
days inactive
4396
days old
devel@lists.crash-utility.osci.io
4 comments
2 participants
participants (2)
-
Dave Anderson -
Oberman, Laurence (GSE MSE)