2:53 a.m.
Hello Dave,
Currently, runq command doesn't show tasks in throttled cfs_rqs/rt_rqs,
but sometimes we want to display them to help us find the problem. So
I made the patches to make them displayed.
Two patches are attached. and PATCH2 should be applied after PATCH1.
PATCH1: 0001-runq-make-tasks-in-cfs_rq-displayed-hierarchically.patch
this patch is used to change the original display of the cfs_rq tasks.
for example, before applying PATCH1, the display is:
CPU 2 RUNQUEUE: ffff88002c316680
CURRENT: PID: 4274 TASK: ffff88013aa2eae0 COMMAND: "sh"
RT PRIO_ARRAY: ffff88002c316808
[no tasks queued]
CFS RB_ROOT: ffff88002c316718
[120] PID: 4296 TASK: ffff88010916d540 COMMAND: "sh"
[120] PID: 4258 TASK: ffff880135c5eaa0 COMMAND: "sh"
[120] PID: 4270 TASK: ffff88013aeb6aa0 COMMAND: "sh"
[120] PID: 4268 TASK: ffff88013b6b4040 COMMAND: "sh"
[120] PID: 4272 TASK: ffff88013a918080 COMMAND: "sh"
[120] PID: 4280 TASK: ffff880109028080 COMMAND: "sh"
[120] PID: 4276 TASK: ffff8801388fa080 COMMAND: "sh"
[120] PID: 4278 TASK: ffff880109029540 COMMAND: "sh"
[120] PID: 4260 TASK: ffff88013aeeeae0 COMMAND: "sh"
[130] PID: 4335 TASK: ffff88011ac3d500 COMMAND: "sosreport"
[120] PID: 4284 TASK: ffff8801050a7540 COMMAND: "sh"
[120] PID: 4286 TASK: ffff8801050a6080 COMMAND: "sh"
[120] PID: 4282 TASK: ffff88011ac48aa0 COMMAND: "sh"
[120] PID: 4290 TASK: ffff8801041f7540 COMMAND: "sh"
[120] PID: 4288 TASK: ffff880121530aa0 COMMAND: "sh"
[120] PID: 4292 TASK: ffff8801041f6080 COMMAND: "sh"
[120] PID: 4262 TASK: ffff880137b21500 COMMAND: "sh"
[120] PID: 4266 TASK: ffff88011aeb4aa0 COMMAND: "sh"
[120] PID: 4264 TASK: ffff88010400b540 COMMAND: "sh"
after applying PATCH1, the display is changed to:
CPU 2 RUNQUEUE: ffff88002c316680
CURRENT: PID: 4274 TASK: ffff88013aa2eae0 COMMAND: "sh"
RT PRIO_ARRAY: ffff88002c316808
[no tasks queued]
CFS RB_ROOT: ffff88002c316718
[120] PID: 4296 TASK: ffff88010916d540 COMMAND: "sh"
[120] PID: 4258 TASK: ffff880135c5eaa0 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff880138f50200 <test1>
GROUP CFS RB_ROOT: ffff88001f8d1e00 <test11>
[120] PID: 4270 TASK: ffff88013aeb6aa0 COMMAND: "sh"
[120] PID: 4268 TASK: ffff88013b6b4040 COMMAND: "sh"
[120] PID: 4272 TASK: ffff88013a918080 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff88013a1ba200 <test12>
[120] PID: 4280 TASK: ffff880109028080 COMMAND: "sh"
[120] PID: 4276 TASK: ffff8801388fa080 COMMAND: "sh"
[120] PID: 4278 TASK: ffff880109029540 COMMAND: "sh"
[120] PID: 4260 TASK: ffff88013aeeeae0 COMMAND: "sh"
[130] PID: 4335 TASK: ffff88011ac3d500 COMMAND: "sosreport"
GROUP CFS RB_ROOT: ffff88013bb75400 <test2>
GROUP CFS RB_ROOT: ffff8801381f1000 <test21>
[120] PID: 4284 TASK: ffff8801050a7540 COMMAND: "sh"
[120] PID: 4286 TASK: ffff8801050a6080 COMMAND: "sh"
[120] PID: 4282 TASK: ffff88011ac48aa0 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff88001f816000 <test22>
[120] PID: 4290 TASK: ffff8801041f7540 COMMAND: "sh"
[120] PID: 4288 TASK: ffff880121530aa0 COMMAND: "sh"
[120] PID: 4292 TASK: ffff8801041f6080 COMMAND: "sh"
[120] PID: 4262 TASK: ffff880137b21500 COMMAND: "sh"
[120] PID: 4266 TASK: ffff88011aeb4aa0 COMMAND: "sh"
[120] PID: 4264 TASK: ffff88010400b540 COMMAND: "sh"
PACTH2: 0002-runq-make-tasks-in-throttled-cfs_rqs-rt_rqs-displaye.patch
this patch is used to display tasks in throttled cfs_rqs and rt_rqs.
for example, before applying PATCH2:
CPU 1 RUNQUEUE: ffff88002c296680
CURRENT: PID: 3494 TASK: ffff88011bcaa080 COMMAND: "sh"
RT PRIO_ARRAY: ffff88002c296808
[no tasks queued]
CFS RB_ROOT: ffff88002c296718
[120] PID: 3516 TASK: ffff88011bfdd500 COMMAND: "sh"
[120] PID: 3514 TASK: ffff88011bef2ae0 COMMAND: "sh"
after applying PATCH2:
CPU 1 RUNQUEUE: ffff88002c296680
CURRENT: PID: 3494 TASK: ffff88011bcaa080 COMMAND: "sh"
RT PRIO_ARRAY: ffff88002c296808
[ 0] GROUP RT PRIO_ARRAY: ffff88013b351800 <test1> (THROTTLED)
[ 0] GROUP RT PRIO_ARRAY: ffff88013acdb800 <test11>
[ 0] PID: 3546 TASK: ffff88010d05e040 COMMAND: "rtloop99"
[ 1] PID: 3541 TASK: ffff88011bef2080 COMMAND: "rtloop98"
[ 54] PID: 3536 TASK: ffff88011bf28080 COMMAND: "rtloop45"
[ 98] PID: 3530 TASK: ffff88013ab1aaa0 COMMAND: "rtloop1"
[ 1] PID: 3540 TASK: ffff88011bf29540 COMMAND: "rtloop98"
[ 54] PID: 3534 TASK: ffff88013ab14040 COMMAND: "rtloop45"
[ 54] GROUP RT PRIO_ARRAY: ffff88013a207800 <test12>
[ 54] PID: 3537 TASK: ffff88013a378ae0 COMMAND: "rtloop45"
[ 98] PID: 3531 TASK: ffff88011bdae080 COMMAND: "rtloop1"
CFS RB_ROOT: ffff88002c296718
[120] PID: 3516 TASK: ffff88011bfdd500 COMMAND: "sh"
[120] PID: 3514 TASK: ffff88011bef2ae0 COMMAND: "sh"
Another example to show the throttled tasks in cfs_rqs:
CPU 1 RUNQUEUE: ffff880028296680
CURRENT: PID: 7814 TASK: ffff880117bab500 COMMAND: "sh"
RT PRIO_ARRAY: ffff880028296808
[no tasks queued]
CFS RB_ROOT: ffff880028296718
[120] PID: 2619 TASK: ffff88013812c040 COMMAND: "Xorg"
......
[120] PID: 7798 TASK: ffff88013a39eae0 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff880135fc6800 <test1> (DEQUEUED)
GROUP CFS RB_ROOT: ffff88000ce5bc00 <test12> (THROTTLED)
[120] PID: 7820 TASK: ffff8801383ab500 COMMAND: "sh"
[120] PID: 7818 TASK: ffff88013ba14ae0 COMMAND: "sh"
[120] PID: 7816 TASK: ffff880117baa040 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff880135fc6800 <test1> (DEQUEUED)
GROUP CFS RB_ROOT: ffff8800b0b32000 <test11> (THROTTLED)
[120] PID: 7810 TASK: ffff8800b0b96aa0 COMMAND: "sh"
*NOTE*
1. The patch also displays the group name that the cfs_rq/rt_rq belows to.
2. There are two statuses about the rqs: DEQUEUED and THROTTLED.
THROTTLED means this cfs_rq/rt_rq is throttled. And DEQUEUED means the
running entities in the rq are 0 because no running task on it and its
child rqs are throttled.
Thanks
Zhang Yanfei
4:05 p.m.
----- Original Message -----
Hello Dave,
Currently, runq command doesn't show tasks in throttled cfs_rqs/rt_rqs,
but sometimes we want to display them to help us find the problem. So
I made the patches to make them displayed.
Two patches are attached. and PATCH2 should be applied after PATCH1.
PATCH1: 0001-runq-make-tasks-in-cfs_rq-displayed-hierarchically.patch
this patch is used to change the original display of the cfs_rq tasks.
for example, before applying PATCH1, the display is:
CPU 2 RUNQUEUE: ffff88002c316680
CURRENT: PID: 4274 TASK: ffff88013aa2eae0 COMMAND: "sh"
RT PRIO_ARRAY: ffff88002c316808
[no tasks queued]
CFS RB_ROOT: ffff88002c316718
[120] PID: 4296 TASK: ffff88010916d540 COMMAND: "sh"
[120] PID: 4258 TASK: ffff880135c5eaa0 COMMAND: "sh"
[120] PID: 4270 TASK: ffff88013aeb6aa0 COMMAND: "sh"
[120] PID: 4268 TASK: ffff88013b6b4040 COMMAND: "sh"
[120] PID: 4272 TASK: ffff88013a918080 COMMAND: "sh"
[120] PID: 4280 TASK: ffff880109028080 COMMAND: "sh"
[120] PID: 4276 TASK: ffff8801388fa080 COMMAND: "sh"
[120] PID: 4278 TASK: ffff880109029540 COMMAND: "sh"
[120] PID: 4260 TASK: ffff88013aeeeae0 COMMAND: "sh"
[130] PID: 4335 TASK: ffff88011ac3d500 COMMAND: "sosreport"
[120] PID: 4284 TASK: ffff8801050a7540 COMMAND: "sh"
[120] PID: 4286 TASK: ffff8801050a6080 COMMAND: "sh"
[120] PID: 4282 TASK: ffff88011ac48aa0 COMMAND: "sh"
[120] PID: 4290 TASK: ffff8801041f7540 COMMAND: "sh"
[120] PID: 4288 TASK: ffff880121530aa0 COMMAND: "sh"
[120] PID: 4292 TASK: ffff8801041f6080 COMMAND: "sh"
[120] PID: 4262 TASK: ffff880137b21500 COMMAND: "sh"
[120] PID: 4266 TASK: ffff88011aeb4aa0 COMMAND: "sh"
[120] PID: 4264 TASK: ffff88010400b540 COMMAND: "sh"
after applying PATCH1, the display is changed to:
CPU 2 RUNQUEUE: ffff88002c316680
CURRENT: PID: 4274 TASK: ffff88013aa2eae0 COMMAND: "sh"
RT PRIO_ARRAY: ffff88002c316808
[no tasks queued]
CFS RB_ROOT: ffff88002c316718
[120] PID: 4296 TASK: ffff88010916d540 COMMAND: "sh"
[120] PID: 4258 TASK: ffff880135c5eaa0 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff880138f50200 <test1>
GROUP CFS RB_ROOT: ffff88001f8d1e00 <test11>
[120] PID: 4270 TASK: ffff88013aeb6aa0 COMMAND: "sh"
[120] PID: 4268 TASK: ffff88013b6b4040 COMMAND: "sh"
[120] PID: 4272 TASK: ffff88013a918080 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff88013a1ba200 <test12>
[120] PID: 4280 TASK: ffff880109028080 COMMAND: "sh"
[120] PID: 4276 TASK: ffff8801388fa080 COMMAND: "sh"
[120] PID: 4278 TASK: ffff880109029540 COMMAND: "sh"
[120] PID: 4260 TASK: ffff88013aeeeae0 COMMAND: "sh"
[130] PID: 4335 TASK: ffff88011ac3d500 COMMAND: "sosreport"
GROUP CFS RB_ROOT: ffff88013bb75400 <test2>
GROUP CFS RB_ROOT: ffff8801381f1000 <test21>
[120] PID: 4284 TASK: ffff8801050a7540 COMMAND: "sh"
[120] PID: 4286 TASK: ffff8801050a6080 COMMAND: "sh"
[120] PID: 4282 TASK: ffff88011ac48aa0 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff88001f816000 <test22>
[120] PID: 4290 TASK: ffff8801041f7540 COMMAND: "sh"
[120] PID: 4288 TASK: ffff880121530aa0 COMMAND: "sh"
[120] PID: 4292 TASK: ffff8801041f6080 COMMAND: "sh"
[120] PID: 4262 TASK: ffff880137b21500 COMMAND: "sh"
[120] PID: 4266 TASK: ffff88011aeb4aa0 COMMAND: "sh"
[120] PID: 4264 TASK: ffff88010400b540 COMMAND: "sh"
PACTH2:
0002-runq-make-tasks-in-throttled-cfs_rqs-rt_rqs-displaye.patch
this patch is used to display tasks in throttled cfs_rqs and rt_rqs.
for example, before applying PATCH2:
CPU 1 RUNQUEUE: ffff88002c296680
CURRENT: PID: 3494 TASK: ffff88011bcaa080 COMMAND: "sh"
RT PRIO_ARRAY: ffff88002c296808
[no tasks queued]
CFS RB_ROOT: ffff88002c296718
[120] PID: 3516 TASK: ffff88011bfdd500 COMMAND: "sh"
[120] PID: 3514 TASK: ffff88011bef2ae0 COMMAND: "sh"
after applying PATCH2:
CPU 1 RUNQUEUE: ffff88002c296680
CURRENT: PID: 3494 TASK: ffff88011bcaa080 COMMAND: "sh"
RT PRIO_ARRAY: ffff88002c296808
[ 0] GROUP RT PRIO_ARRAY: ffff88013b351800 <test1> (THROTTLED)
[ 0] GROUP RT PRIO_ARRAY: ffff88013acdb800 <test11>
[ 0] PID: 3546 TASK: ffff88010d05e040 COMMAND: "rtloop99"
[ 1] PID: 3541 TASK: ffff88011bef2080 COMMAND: "rtloop98"
[ 54] PID: 3536 TASK: ffff88011bf28080 COMMAND: "rtloop45"
[ 98] PID: 3530 TASK: ffff88013ab1aaa0 COMMAND: "rtloop1"
[ 1] PID: 3540 TASK: ffff88011bf29540 COMMAND: "rtloop98"
[ 54] PID: 3534 TASK: ffff88013ab14040 COMMAND: "rtloop45"
[ 54] GROUP RT PRIO_ARRAY: ffff88013a207800 <test12>
[ 54] PID: 3537 TASK: ffff88013a378ae0 COMMAND: "rtloop45"
[ 98] PID: 3531 TASK: ffff88011bdae080 COMMAND: "rtloop1"
CFS RB_ROOT: ffff88002c296718
[120] PID: 3516 TASK: ffff88011bfdd500 COMMAND: "sh"
[120] PID: 3514 TASK: ffff88011bef2ae0 COMMAND: "sh"
Another example to show the throttled tasks in cfs_rqs:
CPU 1 RUNQUEUE: ffff880028296680
CURRENT: PID: 7814 TASK: ffff880117bab500 COMMAND: "sh"
RT PRIO_ARRAY: ffff880028296808
[no tasks queued]
CFS RB_ROOT: ffff880028296718
[120] PID: 2619 TASK: ffff88013812c040 COMMAND: "Xorg"
......
[120] PID: 7798 TASK: ffff88013a39eae0 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff880135fc6800 <test1> (DEQUEUED)
GROUP CFS RB_ROOT: ffff88000ce5bc00 <test12> (THROTTLED)
[120] PID: 7820 TASK: ffff8801383ab500 COMMAND: "sh"
[120] PID: 7818 TASK: ffff88013ba14ae0 COMMAND: "sh"
[120] PID: 7816 TASK: ffff880117baa040 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff880135fc6800 <test1> (DEQUEUED)
GROUP CFS RB_ROOT: ffff8800b0b32000 <test11> (THROTTLED)
[120] PID: 7810 TASK: ffff8800b0b96aa0 COMMAND: "sh"
*NOTE*
1. The patch also displays the group name that the cfs_rq/rt_rq belows to.
2. There are two statuses about the rqs: DEQUEUED and THROTTLED.
THROTTLED means this cfs_rq/rt_rq is throttled. And DEQUEUED means the
running entities in the rq are 0 because no running task on it and its
child rqs are throttled.
Thanks
Zhang Yanfei
Hello Zhang,
I tested your patch against a sample set of dumpfiles as
well as live on my 3.6.2-4.fc17 kernel. Here are my
results and comments.
First, when posting patches, please run the build with
"make warn" in order to catch all these types of complaints:
$ make warn
... [ cut ] ...
cc -c -g -DX86_64 -DGDB_7_3_1 task.c -Wall -O2 -Wstrict-prototypes -Wmissing-prototypes
-fstack-protector
task.c: In function 'dump_task_group_name':
task.c:7545:7: warning: unused variable 'buf' [-Wunused-variable]
task.c: In function 'dump_tasks_in_cfs_rq':
task.c:7584:8: warning: unused variable 'p1' [-Wunused-variable]
task.c: In function 'dump_RT_prio_array':
task.c:8050:8: warning: unused variable 'p1h [-Wunused-variable]
task.c: In function 'cmd_runq':
task.c:8012:31: warning: 'root_task_group' may be used uninitialized in this
function [-Wmaybe-uninitialized]
task.c:7887:8: note: 'root_task_group' was declared here
...
Anyway, when I run this patch on a live Fedora 3.6.2-4.fc17.x86_64
kernel, it shows this *every* time on the active "crash" task:
crash> set
PID: 6825
COMMAND: "crash"
TASK: ffff8801df8bae20 [THREAD_INFO: ffff88020e55a000]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
crash> runq
CPU 0 RUNQUEUE: ffff88021e213cc0
CURRENT: PID: 6825 TASK: ffff8801df8bae20 COMMAND: "crash"
RT PRIO_ARRAY: ffff88021e213e28
[no tasks queued]
CFS RB_ROOT: ffff88021e213d58
GROUP CFS RB_ROOT: ffff8801ebb48200runq: invalid kernel virtual address: 48 type:
"cgroup dentry"
crash>
So clearly that's a problem that needs addressing -- did you
test this patch on a live system?
I see similar issues on vmcores that were taken using the
"snap.so" snapshot extension module. Now, I admit that in
those cases it is certainly possible that the scheduler
infrastructure is undergoing changes *while* the snapshot
was being taken. However, I never see a problem with the
unpatched "runq" command, which always shows the active tasks
correctly. Here are a few examples:
On a 2.6.40.4-5.fc15 snapshot, here is a situation where
using the current runq command, it shows:
CPU 1 RUNQUEUE: ffff88003fc92540
CURRENT: PID: 1341 TASK: ffff880037409730 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc92690
[no tasks queued]
CFS RB_ROOT: ffff88003fc925d8
[no tasks queued]
with your patch, it results in this:
CPU 1 RUNQUEUE: ffff88003fc92540
CURRENT: PID: 1341 TASK: ffff880037409730 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc92690
[no tasks queued]
CFS RB_ROOT: ffff88003fc925d8
[no tasks queued]
[no tasks queued]
On a 2.6.29.4-167.fc11 snapshot, the current runq command shows:
crash> runq
CPU 0 RUNQUEUE: ffff88000101b300
CURRENT: PID: 0 TASK: ffffffff81584360 COMMAND: "swapper"
RT PRIO_ARRAY: ffff88000101b420
[no tasks queued]
CFS RB_ROOT: ffff88000101b398
[no tasks queued]
CPU 1 RUNQUEUE: ffff880001029300
CURRENT: PID: 19625 TASK: ffff8800764e5c00 COMMAND: "crash"
RT PRIO_ARRAY: ffff880001029420
[no tasks queued]
CFS RB_ROOT: ffff880001029398
[no tasks queued]
with your patch, it results in this:
crash> runq
runq: gdb request failed: print &((struct rt_rq *)0x0)->highest_prio.curr
crash>
This is from your patch:
sprintf(buf, "print &((struct rt_rq
*)0x0)->highest_prio.curr");
It is always preferable to use OFFSET() if pre-stored, or at least
use MEMBER_OFFSET() if not, instead of invoking gdb like this. You
already have "offset_table.rt_rq_highest_prio_curr" set up -- why aren't
you using it here? I also saw the same error as above in a 2.6.29.2-52.fc10
snapshot, so perhaps it's a kernel version dependency that you are not
accouting for? In any case, the failure mode above is unacceptable.
Here on another 3.2.6-3.fc16 snapshot, I see this:
CPU 0 RUNQUEUE: ffff88003fc13780
CURRENT: PID: 1383 TASK: ffff88003c932e40 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc13910
[no tasks queued]
CFS RB_ROOT: ffff88003fc13820
[no tasks queued]
With your patch:
CPU 0 RUNQUEUE: ffff88003fc13780
CURRENT: PID: 1383 TASK: ffff88003c932e40 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc13910
[no tasks queued]
CFS RB_ROOT: ffff88003fc13820
GROUP CFS RB_ROOT: ffff88003a432c00runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
So anyway, your patch should be able to at least work as
well on live systems and snapshots as the current runq
command does.
Let me note here that the remainder of the examples below are
from actual crash dumps, i.e. not "snapshots".
First, to be honest about this, I wonder whether the additional task
group data is more, or less, understandable. For example, this example
is from a 2.6.32-220.el6 kernel.
With the current runq:
crash> runq
CPU 0 RUNQUEUE: ffff880133c15fc0
CURRENT: PID: 3144 TASK: ffff88022a446040 COMMAND: "qemu-kvm"
RT PRIO_ARRAY: ffff880133c16148
[no tasks queued]
CFS RB_ROOT: ffff880133c16058
[no tasks queued]
CPU 1 RUNQUEUE: ffff880028215fc0
CURRENT: PID: 2948 TASK: ffff88022af2a100 COMMAND: "bash"
RT PRIO_ARRAY: ffff880028216148
[no tasks queued]
CFS RB_ROOT: ffff880028216058
[120] PID: 3248 TASK: ffff88012a9d4100 COMMAND: "qemu-kvm"
...
With your patch:
crash>
CPU 0 RUNQUEUE: ffff880133c15fc0
CURRENT: PID: 3144 TASK: ffff88022a446040 COMMAND: "qemu-kvm"
RT PRIO_ARRAY: ffff880133c16148
[no tasks queued]
CFS RB_ROOT: ffff880133c16058
GROUP CFS RB_ROOT: ffff88022ac8f000 <libvirt>
GROUP CFS RB_ROOT: ffff88022c075000 <qemu>
GROUP CFS RB_ROOT: ffff88022af94c00 <guest1>
GROUP CFS RB_ROOT: ffff88022aab2600 <vcpu1>
[no tasks queued]
[no tasks queued]
[no tasks queued]
[no tasks queued]
[no tasks queued]
CPU 1 RUNQUEUE: ffff880028215fc0
CURRENT: PID: 2948 TASK: ffff88022af2a100 COMMAND: "bash"
RT PRIO_ARRAY: ffff880028216148
[no tasks queued]
CFS RB_ROOT: ffff880028216058
GROUP CFS RB_ROOT: ffff88012c5d1000 <libvirt>
GROUP CFS RB_ROOT: ffff88012c663e00 <qemu>
GROUP CFS RB_ROOT: ffff88012bb56000 <guest2>
GROUP CFS RB_ROOT: ffff88012b012000 <vcpu0>
[120] PID: 3248 TASK: ffff88012a9d4100 COMMAND:
"qemu-kvm"
...
On CPU 0, there are no other tasks queued, and so the current
runq command shows just that -- whereas your patch shows all of the
other empty RB_ROOT structures. Why bother showing them at all?
And on CPU 1, can you please condense the display a bit? Why does each
GROUP line have to get indented by 6 spaces? Why can't it be indented by
just 3 spaces like the first group?
Here's another 2.6.32-131.0.15.el6 vmcore. With the current runq
command these two cpus look like this:
crash> runq
CPU 0 RUNQUEUE: ffff88000a215f80
CURRENT: PID: 28263 TASK: ffff880037aaa040 COMMAND: "loop.ABA"
RT PRIO_ARRAY: ffff88000a216098
[no tasks queued]
CFS RB_ROOT: ffff88000a216010
[120] PID: 28262 TASK: ffff880037cc40c0 COMMAND: "loop.ABA"
[120] PID: 28271 TASK: ffff8800787a8b40 COMMAND: "loop.ABB"
[120] PID: 28272 TASK: ffff880037afd580 COMMAND: "loop.ABB"
[120] PID: 28245 TASK: ffff8800785e8b00 COMMAND: "loop.AB"
[120] PID: 28246 TASK: ffff880078628ac0 COMMAND: "loop.AB"
[120] PID: 28241 TASK: ffff880078616b40 COMMAND: "loop.AA"
[120] PID: 28239 TASK: ffff8800785774c0 COMMAND: "loop.AA"
[120] PID: 28240 TASK: ffff880078617580 COMMAND: "loop.AA"
[120] PID: 28232 TASK: ffff880079b5d4c0 COMMAND: "loop.A"
... [ cut ] ...
CPU 6 RUNQUEUE: ffff88000a395f80
CURRENT: PID: 28230 TASK: ffff8800373d2b40 COMMAND: "loop.A"
RT PRIO_ARRAY: ffff88000a396098
[no tasks queued]
CFS RB_ROOT: ffff88000a396010
[120] PID: 28270 TASK: ffff88007812ab40 COMMAND: "loop.ABB"
[120] PID: 28261 TASK: ffff880037cc5540 COMMAND: "loop.ABA"
[120] PID: 28244 TASK: ffff88007b4f6a80 COMMAND: "loop.AB"
[120] PID: 28259 TASK: ffff880075978080 COMMAND: "loop.AAB"
[120] PID: 28257 TASK: ffff8800780a0a80 COMMAND: "loop.AAB"
[120] PID: 28258 TASK: ffff880075979500 COMMAND: "loop.AAB"
[120] PID: 28254 TASK: ffff880037d2a040 COMMAND: "loop.AAA"
[120] PID: 28253 TASK: ffff88007b534100 COMMAND: "loop.AAA"
[120] PID: 28255 TASK: ffff880078628080 COMMAND: "loop.AAA"
[120] PID: 28231 TASK: ffff880037b14b40 COMMAND: "loop.A"
...
With your patch:
crash> runq
CPU 0 RUNQUEUE: ffff88000a215f80
CURRENT: PID: 28263 TASK: ffff880037aaa040 COMMAND: "loop.ABA"
RT PRIO_ARRAY: ffff88000a216098
[no tasks queued]
CFS RB_ROOT: ffff88000a216010
[120] PID: 28262 TASK: ffff880037cc40c0 COMMAND:
"loop.ABA"
[120] PID: 28271 TASK: ffff8800787a8b40 COMMAND:
"loop.ABB"
[120] PID: 28272 TASK: ffff880037afd580 COMMAND:
"loop.ABB"
[120] PID: 28245 TASK: ffff8800785e8b00 COMMAND: "loop.AB"
[120] PID: 28246 TASK: ffff880078628ac0 COMMAND: "loop.AB"
[120] PID: 28241 TASK: ffff880078616b40 COMMAND: "loop.AA"
[120] PID: 28239 TASK: ffff8800785774c0 COMMAND: "loop.AA"
[120] PID: 28240 TASK: ffff880078617580 COMMAND: "loop.AA"
[120] PID: 28232 TASK: ffff880079b5d4c0 COMMAND: "loop.A"
... [ cut ] ...
CPU 6 RUNQUEUE: ffff88000a395f80
CURRENT: PID: 28230 TASK: ffff8800373d2b40 COMMAND: "loop.A"
RT PRIO_ARRAY: ffff88000a396098
[no tasks queued]
CFS RB_ROOT: ffff88000a396010
[120] PID: 28270 TASK: ffff88007812ab40 COMMAND:
"loop.ABB"
[120] PID: 28261 TASK: ffff880037cc5540 COMMAND:
"loop.ABA"
[120] PID: 28244 TASK: ffff88007b4f6a80 COMMAND: "loop.AB"
[120] PID: 28259 TASK: ffff880075978080 COMMAND:
"loop.AAB"
[120] PID: 28257 TASK: ffff8800780a0a80 COMMAND:
"loop.AAB"
[120] PID: 28258 TASK: ffff880075979500 COMMAND:
"loop.AAB"
[120] PID: 28254 TASK: ffff880037d2a040 COMMAND:
"loop.AAA"
[120] PID: 28253 TASK: ffff88007b534100 COMMAND:
"loop.AAA"
[120] PID: 28255 TASK: ffff880078628080 COMMAND:
"loop.AAA"
[120] PID: 28231 TASK: ffff880037b14b40 COMMAND: "loop.A"
I'm not sure what the reason is, but that display is clearly unnacceptable.
On a 3.2.1-0.10.el7 vmcore, I see this with the current runq command:
crash> runq
... [ cut ] ...
CPU 3 RUNQUEUE: ffff8804271d43c0
CURRENT: PID: 11615 TASK: ffff88020c50a670 COMMAND: "runtest.sh"
RT PRIO_ARRAY: ffff8804271d4590
[no tasks queued]
CFS RB_ROOT: ffff8804271d44a0
[no tasks queued]
...
With your patch, the command aborts here:
crash> runq
... [ cut ] ...
CPU 3 RUNQUEUE: ffff8804271d43c0
CURRENT: PID: 11615 TASK: ffff88020c50a670 COMMAND: "runtest.sh"
RT PRIO_ARRAY: ffff8804271d4590
[no tasks queued]
CFS RB_ROOT: ffff8804271d44a0
GROUP CFS RB_ROOT: ffff88041e0d2760runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
crash>
And lastly, on a 2.6.27-0.244 I saw this again:
crash> runq
runq: gdb request failed: print &((struct rt_rq *)0x0)->highest_prio.curr
crash>
Let me know if you would like to have any of the vmlinux/vmcore
pairs above. If they are not too huge, I can make them available
for you to download from my people.redhat.com site. Although,
some of the dumpfiles were created and forwarded to me by your
compatriot Daisuke Hatayama... ;-)
And for that matter, I will need for you to make available to me
a sample vmcore that shows the (DEQUEUED) and (THROTTLED) status
so I can have it on hand for testing purposes.
Thanks,
Dave
5:39 a.m.
Hello Dave,
Sorry about not testing the patch fully enough. And I think we
should make a discussion about the first patch. I have done some
tests with the patch, and I attached it. So could you please test
it in your box again.
Hello Zhang,
I tested your patch against a sample set of dumpfiles as
well as live on my 3.6.2-4.fc17 kernel. Here are my
results and comments.
First, when posting patches, please run the build with
"make warn" in order to catch all these types of complaints:
$ make warn
... [ cut ] ...
cc -c -g -DX86_64 -DGDB_7_3_1 task.c -Wall -O2 -Wstrict-prototypes
-Wmissing-prototypes -fstack-protector
task.c: In function 'dump_task_group_name':
task.c:7545:7: warning: unused variable 'buf' [-Wunused-variable]
task.c: In function 'dump_tasks_in_cfs_rq':
task.c:7584:8: warning: unused variable 'p1' [-Wunused-variable]
task.c: In function 'dump_RT_prio_array':
task.c:8050:8: warning: unused variable 'p1h [-Wunused-variable]
task.c: In function 'cmd_runq':
task.c:8012:31: warning: 'root_task_group' may be used uninitialized in this
function [-Wmaybe-uninitialized]
task.c:7887:8: note: 'root_task_group' was declared here
...
my neglect.
Anyway, when I run this patch on a live Fedora 3.6.2-4.fc17.x86_64
kernel, it shows this *every* time on the active "crash" task:
crash> set
PID: 6825
COMMAND: "crash"
TASK: ffff8801df8bae20 [THREAD_INFO: ffff88020e55a000]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
crash> runq
CPU 0 RUNQUEUE: ffff88021e213cc0
CURRENT: PID: 6825 TASK: ffff8801df8bae20 COMMAND: "crash"
RT PRIO_ARRAY: ffff88021e213e28
[no tasks queued]
CFS RB_ROOT: ffff88021e213d58
GROUP CFS RB_ROOT: ffff8801ebb48200runq: invalid kernel virtual address: 48 type:
"cgroup dentry"
crash>
So clearly that's a problem that needs addressing -- did you
test this patch on a live system?
I see similar issues on vmcores that were taken using the
"snap.so" snapshot extension module. Now, I admit that in
those cases it is certainly possible that the scheduler
infrastructure is undergoing changes *while* the snapshot
was being taken. However, I never see a problem with the
unpatched "runq" command, which always shows the active tasks
correctly. Here are a few examples:
On a 2.6.40.4-5.fc15 snapshot, here is a situation where
using the current runq command, it shows:
CPU 1 RUNQUEUE: ffff88003fc92540
CURRENT: PID: 1341 TASK: ffff880037409730 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc92690
[no tasks queued]
CFS RB_ROOT: ffff88003fc925d8
[no tasks queued]
with your patch, it results in this:
CPU 1 RUNQUEUE: ffff88003fc92540
CURRENT: PID: 1341 TASK: ffff880037409730 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc92690
[no tasks queued]
CFS RB_ROOT: ffff88003fc925d8
[no tasks queued]
[no tasks queued]
This is fixed in the new patch, I think.
On a 2.6.29.4-167.fc11 snapshot, the current runq command shows:
crash> runq
CPU 0 RUNQUEUE: ffff88000101b300
CURRENT: PID: 0 TASK: ffffffff81584360 COMMAND: "swapper"
RT PRIO_ARRAY: ffff88000101b420
[no tasks queued]
CFS RB_ROOT: ffff88000101b398
[no tasks queued]
CPU 1 RUNQUEUE: ffff880001029300
CURRENT: PID: 19625 TASK: ffff8800764e5c00 COMMAND: "crash"
RT PRIO_ARRAY: ffff880001029420
[no tasks queued]
CFS RB_ROOT: ffff880001029398
[no tasks queued]
with your patch, it results in this:
crash> runq
runq: gdb request failed: print &((struct rt_rq *)0x0)->highest_prio.curr
crash>
will be fixed in patch2 later.
This is from your patch:
> sprintf(buf, "print &((struct rt_rq
*)0x0)->highest_prio.curr");
It is always preferable to use OFFSET() if pre-stored, or at least
use MEMBER_OFFSET() if not, instead of invoking gdb like this. You
already have "offset_table.rt_rq_highest_prio_curr" set up -- why aren't
you using it here? I also saw the same error as above in a 2.6.29.2-52.fc10
snapshot, so perhaps it's a kernel version dependency that you are not
accouting for? In any case, the failure mode above is unacceptable.
Here on another 3.2.6-3.fc16 snapshot, I see this:
CPU 0 RUNQUEUE: ffff88003fc13780
CURRENT: PID: 1383 TASK: ffff88003c932e40 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc13910
[no tasks queued]
CFS RB_ROOT: ffff88003fc13820
[no tasks queued]
With your patch:
CPU 0 RUNQUEUE: ffff88003fc13780
CURRENT: PID: 1383 TASK: ffff88003c932e40 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc13910
[no tasks queued]
CFS RB_ROOT: ffff88003fc13820
GROUP CFS RB_ROOT: ffff88003a432c00runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
This is fixed in the new patch, I think.
So anyway, your patch should be able to at least work as
well on live systems and snapshots as the current runq
command does.
Let me note here that the remainder of the examples below are
from actual crash dumps, i.e. not "snapshots".
First, to be honest about this, I wonder whether the additional task
group data is more, or less, understandable. For example, this example
is from a 2.6.32-220.el6 kernel.
With the current runq:
crash> runq
CPU 0 RUNQUEUE: ffff880133c15fc0
CURRENT: PID: 3144 TASK: ffff88022a446040 COMMAND: "qemu-kvm"
RT PRIO_ARRAY: ffff880133c16148
[no tasks queued]
CFS RB_ROOT: ffff880133c16058
[no tasks queued]
CPU 1 RUNQUEUE: ffff880028215fc0
CURRENT: PID: 2948 TASK: ffff88022af2a100 COMMAND: "bash"
RT PRIO_ARRAY: ffff880028216148
[no tasks queued]
CFS RB_ROOT: ffff880028216058
[120] PID: 3248 TASK: ffff88012a9d4100 COMMAND: "qemu-kvm"
...
With your patch:
crash>
CPU 0 RUNQUEUE: ffff880133c15fc0
CURRENT: PID: 3144 TASK: ffff88022a446040 COMMAND: "qemu-kvm"
RT PRIO_ARRAY: ffff880133c16148
[no tasks queued]
CFS RB_ROOT: ffff880133c16058
GROUP CFS RB_ROOT: ffff88022ac8f000 <libvirt>
GROUP CFS RB_ROOT: ffff88022c075000 <qemu>
GROUP CFS RB_ROOT: ffff88022af94c00 <guest1>
GROUP CFS RB_ROOT: ffff88022aab2600 <vcpu1>
[no tasks queued]
[no tasks queued]
[no tasks queued]
[no tasks queued]
[no tasks queued]
This condition happens when a cfs_rq has only one task and the task is
chosen "current", so the only task is dequeued from the cfs_rq, but the
cfs_rq is still in its parent cfs_rq. Just like the above display:
vcpu1 has only one task named qemu-kvm, and this task is current. vcpu1
is still in guest1.
I have changed the display in the new patch,
and now it displays:
crash>
CPU 0 RUNQUEUE: ffff880133c15fc0
CURRENT: PID: 3144 TASK: ffff88022a446040 COMMAND: "qemu-kvm"
RT PRIO_ARRAY: ffff880133c16148
[no tasks queued]
CFS RB_ROOT: ffff880133c16058
GROUP CFS RB_ROOT: ffff88022ac8f000 <libvirt>
GROUP CFS RB_ROOT: ffff88022c075000 <qemu>
GROUP CFS RB_ROOT: ffff88022af94c00 <guest1>
GROUP CFS RB_ROOT: ffff88022aab2600 <vcpu1>
[no tasks queued]
how do you think about this? The reason that I still display the RB_ROOT, like
libvirt, qemu, guest1, is they are entities in their parent cfs_rq, just like
the task.
CPU 1 RUNQUEUE: ffff880028215fc0
CURRENT: PID: 2948 TASK: ffff88022af2a100 COMMAND: "bash"
RT PRIO_ARRAY: ffff880028216148
[no tasks queued]
CFS RB_ROOT: ffff880028216058
GROUP CFS RB_ROOT: ffff88012c5d1000 <libvirt>
GROUP CFS RB_ROOT: ffff88012c663e00 <qemu>
GROUP CFS RB_ROOT: ffff88012bb56000 <guest2>
GROUP CFS RB_ROOT: ffff88012b012000 <vcpu0>
[120] PID: 3248 TASK: ffff88012a9d4100 COMMAND:
"qemu-kvm"
...
On CPU 0, there are no other tasks queued, and so the current
runq command shows just that -- whereas your patch shows all of the
other empty RB_ROOT structures. Why bother showing them at all?
And on CPU 1, can you please condense the display a bit? Why does each
GROUP line have to get indented by 6 spaces? Why can't it be indented by
just 3 spaces like the first group?
OK, this is changed.
Here's another 2.6.32-131.0.15.el6 vmcore. With the current runq
command these two cpus look like this:
crash> runq
CPU 0 RUNQUEUE: ffff88000a215f80
CURRENT: PID: 28263 TASK: ffff880037aaa040 COMMAND: "loop.ABA"
RT PRIO_ARRAY: ffff88000a216098
[no tasks queued]
CFS RB_ROOT: ffff88000a216010
[120] PID: 28262 TASK: ffff880037cc40c0 COMMAND: "loop.ABA"
[120] PID: 28271 TASK: ffff8800787a8b40 COMMAND: "loop.ABB"
[120] PID: 28272 TASK: ffff880037afd580 COMMAND: "loop.ABB"
[120] PID: 28245 TASK: ffff8800785e8b00 COMMAND: "loop.AB"
[120] PID: 28246 TASK: ffff880078628ac0 COMMAND: "loop.AB"
[120] PID: 28241 TASK: ffff880078616b40 COMMAND: "loop.AA"
[120] PID: 28239 TASK: ffff8800785774c0 COMMAND: "loop.AA"
[120] PID: 28240 TASK: ffff880078617580 COMMAND: "loop.AA"
[120] PID: 28232 TASK: ffff880079b5d4c0 COMMAND: "loop.A"
... [ cut ] ...
CPU 6 RUNQUEUE: ffff88000a395f80
CURRENT: PID: 28230 TASK: ffff8800373d2b40 COMMAND: "loop.A"
RT PRIO_ARRAY: ffff88000a396098
[no tasks queued]
CFS RB_ROOT: ffff88000a396010
[120] PID: 28270 TASK: ffff88007812ab40 COMMAND: "loop.ABB"
[120] PID: 28261 TASK: ffff880037cc5540 COMMAND: "loop.ABA"
[120] PID: 28244 TASK: ffff88007b4f6a80 COMMAND: "loop.AB"
[120] PID: 28259 TASK: ffff880075978080 COMMAND: "loop.AAB"
[120] PID: 28257 TASK: ffff8800780a0a80 COMMAND: "loop.AAB"
[120] PID: 28258 TASK: ffff880075979500 COMMAND: "loop.AAB"
[120] PID: 28254 TASK: ffff880037d2a040 COMMAND: "loop.AAA"
[120] PID: 28253 TASK: ffff88007b534100 COMMAND: "loop.AAA"
[120] PID: 28255 TASK: ffff880078628080 COMMAND: "loop.AAA"
[120] PID: 28231 TASK: ffff880037b14b40 COMMAND: "loop.A"
...
With your patch:
crash> runq
CPU 0 RUNQUEUE: ffff88000a215f80
CURRENT: PID: 28263 TASK: ffff880037aaa040 COMMAND: "loop.ABA"
RT PRIO_ARRAY: ffff88000a216098
[no tasks queued]
CFS RB_ROOT: ffff88000a216010
[120] PID: 28262 TASK: ffff880037cc40c0 COMMAND:
"loop.ABA"
[120] PID: 28271 TASK: ffff8800787a8b40 COMMAND:
"loop.ABB"
[120] PID: 28272 TASK: ffff880037afd580 COMMAND:
"loop.ABB"
[120] PID: 28245 TASK: ffff8800785e8b00 COMMAND: "loop.AB"
[120] PID: 28246 TASK: ffff880078628ac0 COMMAND: "loop.AB"
[120] PID: 28241 TASK: ffff880078616b40 COMMAND: "loop.AA"
[120] PID: 28239 TASK: ffff8800785774c0 COMMAND: "loop.AA"
[120] PID: 28240 TASK: ffff880078617580 COMMAND: "loop.AA"
[120] PID: 28232 TASK: ffff880079b5d4c0 COMMAND: "loop.A"
... [ cut ] ...
CPU 6 RUNQUEUE: ffff88000a395f80
CURRENT: PID: 28230 TASK: ffff8800373d2b40 COMMAND: "loop.A"
RT PRIO_ARRAY: ffff88000a396098
[no tasks queued]
CFS RB_ROOT: ffff88000a396010
[120] PID: 28270 TASK: ffff88007812ab40 COMMAND:
"loop.ABB"
[120] PID: 28261 TASK: ffff880037cc5540 COMMAND:
"loop.ABA"
[120] PID: 28244 TASK: ffff88007b4f6a80 COMMAND: "loop.AB"
[120] PID: 28259 TASK: ffff880075978080 COMMAND:
"loop.AAB"
[120] PID: 28257 TASK: ffff8800780a0a80 COMMAND:
"loop.AAB"
[120] PID: 28258 TASK: ffff880075979500 COMMAND:
"loop.AAB"
[120] PID: 28254 TASK: ffff880037d2a040 COMMAND:
"loop.AAA"
[120] PID: 28253 TASK: ffff88007b534100 COMMAND:
"loop.AAA"
[120] PID: 28255 TASK: ffff880078628080 COMMAND:
"loop.AAA"
[120] PID: 28231 TASK: ffff880037b14b40 COMMAND: "loop.A"
I'm not sure what the reason is, but that display is clearly unnacceptable.
This is fixed in the new patch, I think.
On a 3.2.1-0.10.el7 vmcore, I see this with the current runq command:
crash> runq
... [ cut ] ...
CPU 3 RUNQUEUE: ffff8804271d43c0
CURRENT: PID: 11615 TASK: ffff88020c50a670 COMMAND: "runtest.sh"
RT PRIO_ARRAY: ffff8804271d4590
[no tasks queued]
CFS RB_ROOT: ffff8804271d44a0
[no tasks queued]
...
With your patch, the command aborts here:
crash> runq
... [ cut ] ...
CPU 3 RUNQUEUE: ffff8804271d43c0
CURRENT: PID: 11615 TASK: ffff88020c50a670 COMMAND: "runtest.sh"
RT PRIO_ARRAY: ffff8804271d4590
[no tasks queued]
CFS RB_ROOT: ffff8804271d44a0
GROUP CFS RB_ROOT: ffff88041e0d2760runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
crash>
This is fixed in the new patch, I think.
And lastly, on a 2.6.27-0.244 I saw this again:
crash> runq
runq: gdb request failed: print &((struct rt_rq *)0x0)->highest_prio.curr
crash>
will be fixed in patch2 later.
Let me know if you would like to have any of the vmlinux/vmcore
pairs above. If they are not too huge, I can make them available
for you to download from my people.redhat.com site. Although,
some of the dumpfiles were created and forwarded to me by your
compatriot Daisuke Hatayama... ;-)
And for that matter, I will need for you to make available to me
a sample vmcore that shows the (DEQUEUED) and (THROTTLED) status
so I can have it on hand for testing purposes.
I will give you the sample after we finish the first patch, ok?
Thanks
Zhang
1:26 p.m.
----- Original Message -----
Hello Dave,
Sorry about not testing the patch fully enough. And I think we
should make a discussion about the first patch. I have done some
tests with the patch, and I attached it. So could you please test
it in your box again.
Hello Zhang,
I applied only your new patch1 (the old patch2 no longer applies
after this new patch1), and I see this:
$ make warn
...
cc -c -g -DX86_64 -DGDB_7_3_1 task.c -Wall -O2 -Wstrict-prototypes
-Wmissing-prototypes -fstack-protector
task.c: In function ‘dump_CFS_runqueues’:
task.c:7693:6: warning: variable 'tot' set but not used
[-Wunused-but-set-variable]
...
And I still (always) see the same problem with a live kernel:
crash> set
PID: 25998
COMMAND: "crash"
TASK: ffff88020fd9dc40 [THREAD_INFO: ffff88017b6d2000]
CPU: 2
STATE: TASK_RUNNING (ACTIVE)
crash> runq
CPU 0 RUNQUEUE: ffff88021e213cc0
CURRENT: PID: 0 TASK: ffffffff81c13420 COMMAND: "swapper/0"
RT PRIO_ARRAY: ffff88021e213e28
[no tasks queued]
CFS RB_ROOT: ffff88021e213d58
GROUP CFS RB_ROOT: ffff88020ec3b800runq: invalid kernel virtual address: 48 type:
"cgroup dentry"
crash>
I also still see numerous instances of the error above with some
(but not all) of my "snapshot" dumpfiles, where your dump_task_group_name()
function is encountering (and trying to use) a NULL cgroup address here:
static void
dump_task_group_name(ulong group)
{
ulong cgroup, dentry, name;
char *dentry_buf;
int len;
char tmp_buf[100];
readmem(group + OFFSET(task_group_css) + OFFSET(cgroup_subsys_state_cgroup),
KVADDR, &cgroup, sizeof(ulong),
"task_group css cgroup", FAULT_ON_ERROR);
readmem(cgroup + OFFSET(cgroup_dentry), KVADDR, &dentry, sizeof(ulong),
"cgroup dentry", FAULT_ON_ERROR);
Here are the examples, where it always happens on the "crash" process while
it's performing the snapshot file creation:
2.6.38.2-9.fc15 snapshot:
crash> runq
CPU 0 RUNQUEUE: ffff88003fc13840
CURRENT: PID: 1180 TASK: ffff88003bea2e40 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc13988
[no tasks queued]
CFS RB_ROOT: ffff88003fc138d8
GROUP CFS RB_ROOT: ffff880037ef1b00runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
crash>
2.6.40.4-5.fc15 snapshot:
crash> runq
...
CPU 1 RUNQUEUE: ffff88003fc92540
CURRENT: PID: 1341 TASK: ffff880037409730 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc92690
[no tasks queued]
CFS RB_ROOT: ffff88003fc925d8
GROUP CFS RB_ROOT: ffff880037592f00runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
crash>
3.5.1-1.fc17 snapshot:
crash> runq
...
CPU 1 RUNQUEUE: ffff88003ed13800
CURRENT: PID: 31736 TASK: ffff88007c46ae20 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003ed13968
[no tasks queued]
CFS RB_ROOT: ffff88003ed13898
GROUP CFS RB_ROOT: ffff88003deb3000runq: invalid kernel virtual address: 48 type:
"cgroup dentry"
crash>
3.1.7-1.fc16 snapshot:
crash> runq
...
CPU 2 RUNQUEUE: ffff88003e253180
CURRENT: PID: 1495 TASK: ffff880037a60000 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003e2532d0
[no tasks queued]
CFS RB_ROOT: ffff88003e253218
GROUP CFS RB_ROOT: ffff8800277f8500runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
crash>
3.2.6-3.fc16 snapshot:
crash> runq
...
CPU 0 RUNQUEUE: ffff88003fc13780
CURRENT: PID: 1383 TASK: ffff88003c932e40 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc13910
[no tasks queued]
CFS RB_ROOT: ffff88003fc13820
GROUP CFS RB_ROOT: ffff88003a432c00runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
crash>
But I also saw the error above on this 3.2.1-0.8.el7.x86_64 kernel
that actually crashed:
crash> runq
...
CPU 3 RUNQUEUE: ffff8804271d43c0
CURRENT: PID: 11615 TASK: ffff88020c50a670 COMMAND: "runtest.sh"
RT PRIO_ARRAY: ffff8804271d4590
[no tasks queued]
CFS RB_ROOT: ffff8804271d44a0
GROUP CFS RB_ROOT: ffff88041e0d2760runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
crash>
will be fixed in patch2 later.
With respect to your patch2:
+#define MAX_THROTTLED_RQ 100
+struct throttled_rq {
+ ulong rq;
+ int depth;
+ int prio;
+};
+static struct throttled_rq throttled_rt_rq_array[MAX_THROTTLED_RQ];
+static struct throttled_rq throttled_cfs_rq_array[MAX_THROTTLED_RQ];
Can you please dynamically allocate the throttled_rt_rq_array and
throttled_cfs_rq_array arrays with GETBUF(), perhaps in the
task_group_offset_init() function? They are only needed when
"runq" is executed, and then only if the kernel version supports
them. You can FREEBUF() them at the bottom of dump_CFS_runqueues(),
and if the command fails prematurely, they will be FREEBUF()'d
automatically by restore_sanity().
But this leads to the larger question of showing the task_group
data. Consider that the current "runq" command does what it says
it does:
crash> help runq
NAME
runq - run queue
SYNOPSIS
runq [-t]
DESCRIPTION
With no argument, this command displays the tasks on the run queues
of each cpu.
-t Display the timestamp information of each cpu's runqueue, which is the
rq.clock, rq.most_recent_timestamp or rq.timestamp_last_tick value,
whichever applies; following each cpu timestamp is the last_run or
timestamp value of the active task on that cpu, whichever applies,
along with the task identification.
...
Now, your patch adds signficant complexity to the runq handling code
and to its future maintainability. I'm wondering whether your patch
can be modified such that the task_group info would only be displayed
via a new flag, let's say "runq -g". It seems that there has been
considerable churn in the kernel code in this area, and it worries me
that this patch will potentially and unnecessarily cause the breakage
of the simple display of the queued tasks.
Thanks,
Dave
3:48 a.m.
Hello Dave,
于 2012年10月26日 02:26, Dave Anderson 写道:
----- Original Message -----
> Hello Dave,
>
> Sorry about not testing the patch fully enough. And I think we
> should make a discussion about the first patch. I have done some
> tests with the patch, and I attached it. So could you please test
> it in your box again.
Hello Zhang,
I applied only your new patch1 (the old patch2 no longer applies
after this new patch1), and I see this:
$ make warn
...
cc -c -g -DX86_64 -DGDB_7_3_1 task.c -Wall -O2 -Wstrict-prototypes
-Wmissing-prototypes -fstack-protector
task.c: In function ‘dump_CFS_runqueues’:
task.c:7693:6: warning: variable 'tot' set but not used
[-Wunused-but-set-variable]
...
Very sorry about this, it is strange that I didn't find it.
And I still (always) see the same problem with a live kernel:
crash> set
PID: 25998
COMMAND: "crash"
TASK: ffff88020fd9dc40 [THREAD_INFO: ffff88017b6d2000]
CPU: 2
STATE: TASK_RUNNING (ACTIVE)
crash> runq
CPU 0 RUNQUEUE: ffff88021e213cc0
CURRENT: PID: 0 TASK: ffffffff81c13420 COMMAND: "swapper/0"
RT PRIO_ARRAY: ffff88021e213e28
[no tasks queued]
CFS RB_ROOT: ffff88021e213d58
GROUP CFS RB_ROOT: ffff88020ec3b800runq: invalid kernel virtual address: 48 type:
"cgroup dentry"
crash>
I also still see numerous instances of the error above with some
(but not all) of my "snapshot" dumpfiles, where your dump_task_group_name()
function is encountering (and trying to use) a NULL cgroup address here:
I tested the patch on RHEL7.0ALPHA2, and I think it is the autogroup thing.
So I add the cgroup check in the patch, it cgroup is 0, function dump_task_group_name will
just return.
static void
dump_task_group_name(ulong group)
{
ulong cgroup, dentry, name;
char *dentry_buf;
int len;
char tmp_buf[100];
readmem(group + OFFSET(task_group_css) + OFFSET(cgroup_subsys_state_cgroup),
KVADDR, &cgroup, sizeof(ulong),
"task_group css cgroup", FAULT_ON_ERROR);
readmem(cgroup + OFFSET(cgroup_dentry), KVADDR, &dentry, sizeof(ulong),
"cgroup dentry", FAULT_ON_ERROR);
Here are the examples, where it always happens on the "crash" process while
it's performing the snapshot file creation:
2.6.38.2-9.fc15 snapshot:
crash> runq
CPU 0 RUNQUEUE: ffff88003fc13840
CURRENT: PID: 1180 TASK: ffff88003bea2e40 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc13988
[no tasks queued]
CFS RB_ROOT: ffff88003fc138d8
GROUP CFS RB_ROOT: ffff880037ef1b00runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
crash>
2.6.40.4-5.fc15 snapshot:
crash> runq
...
CPU 1 RUNQUEUE: ffff88003fc92540
CURRENT: PID: 1341 TASK: ffff880037409730 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc92690
[no tasks queued]
CFS RB_ROOT: ffff88003fc925d8
GROUP CFS RB_ROOT: ffff880037592f00runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
crash>
3.5.1-1.fc17 snapshot:
crash> runq
...
CPU 1 RUNQUEUE: ffff88003ed13800
CURRENT: PID: 31736 TASK: ffff88007c46ae20 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003ed13968
[no tasks queued]
CFS RB_ROOT: ffff88003ed13898
GROUP CFS RB_ROOT: ffff88003deb3000runq: invalid kernel virtual address: 48 type:
"cgroup dentry"
crash>
3.1.7-1.fc16 snapshot:
crash> runq
...
CPU 2 RUNQUEUE: ffff88003e253180
CURRENT: PID: 1495 TASK: ffff880037a60000 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003e2532d0
[no tasks queued]
CFS RB_ROOT: ffff88003e253218
GROUP CFS RB_ROOT: ffff8800277f8500runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
crash>
3.2.6-3.fc16 snapshot:
crash> runq
...
CPU 0 RUNQUEUE: ffff88003fc13780
CURRENT: PID: 1383 TASK: ffff88003c932e40 COMMAND: "crash"
RT PRIO_ARRAY: ffff88003fc13910
[no tasks queued]
CFS RB_ROOT: ffff88003fc13820
GROUP CFS RB_ROOT: ffff88003a432c00runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
crash>
But I also saw the error above on this 3.2.1-0.8.el7.x86_64 kernel
that actually crashed:
crash> runq
...
CPU 3 RUNQUEUE: ffff8804271d43c0
CURRENT: PID: 11615 TASK: ffff88020c50a670 COMMAND: "runtest.sh"
RT PRIO_ARRAY: ffff8804271d4590
[no tasks queued]
CFS RB_ROOT: ffff8804271d44a0
GROUP CFS RB_ROOT: ffff88041e0d2760runq: invalid kernel virtual address: 38 type:
"cgroup dentry"
crash>
> will be fixed in patch2 later.
With respect to your patch2:
+#define MAX_THROTTLED_RQ 100
+struct throttled_rq {
+ ulong rq;
+ int depth;
+ int prio;
+};
+static struct throttled_rq throttled_rt_rq_array[MAX_THROTTLED_RQ];
+static struct throttled_rq throttled_cfs_rq_array[MAX_THROTTLED_RQ];
Can you please dynamically allocate the throttled_rt_rq_array and
throttled_cfs_rq_array arrays with GETBUF(), perhaps in the
task_group_offset_init() function? They are only needed when
"runq" is executed, and then only if the kernel version supports
them. You can FREEBUF() them at the bottom of dump_CFS_runqueues(),
and if the command fails prematurely, they will be FREEBUF()'d
automatically by restore_sanity().
OK.
But this leads to the larger question of showing the task_group
data. Consider that the current "runq" command does what it says
it does:
crash> help runq
NAME
runq - run queue
SYNOPSIS
runq [-t]
DESCRIPTION
With no argument, this command displays the tasks on the run queues
of each cpu.
-t Display the timestamp information of each cpu's runqueue, which is the
rq.clock, rq.most_recent_timestamp or rq.timestamp_last_tick value,
whichever applies; following each cpu timestamp is the last_run or
timestamp value of the active task on that cpu, whichever applies,
along with the task identification.
...
Now, your patch adds signficant complexity to the runq handling code
and to its future maintainability. I'm wondering whether your patch
can be modified such that the task_group info would only be displayed
via a new flag, let's say "runq -g". It seems that there has been
considerable churn in the kernel code in this area, and it worries me
that this patch will potentially and unnecessarily cause the breakage
of the simple display of the queued tasks.
Currently, rt_rq is displayed hierarchically, while the cfs_rq is displayed
like that. So I made the first patch to make tasks in cfs_rq displayed
hierarchically so that we could see which task belongs to which cfs_rq
easily, just like rt_rq.
The second patch is used to display tasks in throttled cfs_rqs/rt_rqs.
To display tasks in throttled cfs_rqs/rt_rqs is easy, but to display them
sorted in the current runqueue is kind of difficult, so the patch2 looks
complex. I think I will implement the patch2 by two ways, one is just the fix
version of the current patch2, the other is to use the new flag '-g'.
You can decide which one to apply.
The attachment is the patch1.
Thanks
Zhang
2:05 p.m.
----- Original Message -----
>
> Now, your patch adds signficant complexity to the runq handling code
> and to its future maintainability. I'm wondering whether your patch
> can be modified such that the task_group info would only be displayed
> via a new flag, let's say "runq -g". It seems that there has been
> considerable churn in the kernel code in this area, and it worries me
> that this patch will potentially and unnecessarily cause the breakage
> of the simple display of the queued tasks.
>
Currently, rt_rq is displayed hierarchically, while the cfs_rq is displayed
like that. So I made the first patch to make tasks in cfs_rq displayed
hierarchically so that we could see which task belongs to which cfs_rq
easily, just like rt_rq.
The second patch is used to display tasks in throttled cfs_rqs/rt_rqs.
To display tasks in throttled cfs_rqs/rt_rqs is easy, but to display them
sorted in the current runqueue is kind of difficult, so the patch2 looks
complex. I think I will implement the patch2 by two ways, one is just the fix
version of the current patch2, the other is to use the new flag '-g'.
You can decide which one to apply.
The attachment is the patch1.
The latest patch1 seems to test OK, although your change to dump_task_runq_entry()
that removes the leading spaces breaks the output format of "runq -d". But
adding
an INDENT(5) to dump_on_rq_tasks() fixes that.
My suggestion re: a new -g flag would have it applicable to both CFS and RT
run queues (even though the RT groups are currently being shown). I was
thinking that it might be safest to have "runq" (with no argument) to just
display a simple list of tasks, and only show group information with -g.
Are you suggesting that -g would only apply to throttled cfs_rqs/rt_rqs?
Dave
1:06 a.m.
于 2012年10月30日 03:05, Dave Anderson 写道:
----- Original Message -----
>>
>> Now, your patch adds signficant complexity to the runq handling code
>> and to its future maintainability. I'm wondering whether your patch
>> can be modified such that the task_group info would only be displayed
>> via a new flag, let's say "runq -g". It seems that there has been
>> considerable churn in the kernel code in this area, and it worries me
>> that this patch will potentially and unnecessarily cause the breakage
>> of the simple display of the queued tasks.
>>
>
> Currently, rt_rq is displayed hierarchically, while the cfs_rq is displayed
> like that. So I made the first patch to make tasks in cfs_rq displayed
> hierarchically so that we could see which task belongs to which cfs_rq
> easily, just like rt_rq.
>
> The second patch is used to display tasks in throttled cfs_rqs/rt_rqs.
> To display tasks in throttled cfs_rqs/rt_rqs is easy, but to display them
> sorted in the current runqueue is kind of difficult, so the patch2 looks
> complex. I think I will implement the patch2 by two ways, one is just the fix
> version of the current patch2, the other is to use the new flag '-g'.
> You can decide which one to apply.
>
> The attachment is the patch1.
The latest patch1 seems to test OK, although your change to dump_task_runq_entry()
that removes the leading spaces breaks the output format of "runq -d". But
adding
an INDENT(5) to dump_on_rq_tasks() fixes that.
My suggestion re: a new -g flag would have it applicable to both CFS and RT
run queues (even though the RT groups are currently being shown). I was
thinking that it might be safest to have "runq" (with no argument) to just
display a simple list of tasks, and only show group information with -g.
Are you suggesting that -g would only apply to throttled cfs_rqs/rt_rqs?
nop, '-g' is ok to show all rqs with group information.
I rewrote the patch by adding option '-g' to runq and not changing any behaviour
of
the runq (with no argument).
TODO:
1. The help info about the -g option.
I still have a suggestion: when we dump cfs_rq, we don't dump it hierarchically,
but for the current rt_rq, we dump it hierarchically with group info. So could I
make another patch to remove the rt_rq hierarchy, just make it displayed identical
with cfs_rq display.(For -g will display group info for both rt_rq and cfs_rq)
Thanks
Zhang
8:12 a.m.
----- Original Message -----
于 2012年10月30日 03:05, Dave Anderson 写道:
>
>
> ----- Original Message -----
>
>>>
>>> Now, your patch adds signficant complexity to the runq handling
>>> code
>>> and to its future maintainability. I'm wondering whether your
>>> patch
>>> can be modified such that the task_group info would only be
>>> displayed
>>> via a new flag, let's say "runq -g". It seems that there has
>>> been
>>> considerable churn in the kernel code in this area, and it
>>> worries me
>>> that this patch will potentially and unnecessarily cause the
>>> breakage
>>> of the simple display of the queued tasks.
>>>
>>
>> Currently, rt_rq is displayed hierarchically, while the cfs_rq is
>> displayed
>> like that. So I made the first patch to make tasks in cfs_rq
>> displayed
>> hierarchically so that we could see which task belongs to which
>> cfs_rq
>> easily, just like rt_rq.
>>
>> The second patch is used to display tasks in throttled
>> cfs_rqs/rt_rqs.
>> To display tasks in throttled cfs_rqs/rt_rqs is easy, but to
>> display them
>> sorted in the current runqueue is kind of difficult, so the patch2
>> looks
>> complex. I think I will implement the patch2 by two ways, one is
>> just the fix
>> version of the current patch2, the other is to use the new flag
>> '-g'.
>> You can decide which one to apply.
>>
>> The attachment is the patch1.
>
> The latest patch1 seems to test OK, although your change to dump_task_runq_entry()
> that removes the leading spaces breaks the output format of "runq -d".
But adding
> an INDENT(5) to dump_on_rq_tasks() fixes that.
>
> My suggestion re: a new -g flag would have it applicable to both CFS and RT
> run queues (even though the RT groups are currently being shown). I was
> thinking that it might be safest to have "runq" (with no argument) to
just
> display a simple list of tasks, and only show group information with -g.
> Are you suggesting that -g would only apply to throttled cfs_rqs/rt_rqs?
>
nop, '-g' is ok to show all rqs with group information.
I rewrote the patch by adding option '-g' to runq and not changing any behaviour
of
the runq (with no argument).
TODO:
1. The help info about the -g option.
I still have a suggestion: when we dump cfs_rq, we don't dump it hierarchically,
but for the current rt_rq, we dump it hierarchically with group info.
So could I make another patch to remove the rt_rq hierarchy, just make it
displayed identical with cfs_rq display.(For -g will display group info for both rt_rq
and cfs_rq)
Yes, that's exactly what I was suggesting.
I worry about the large number of kernel structure.member declarations that the
command depends upon, because if just one of them changes, it breaks the command.
So it would be preferable if the "runq" command (with no arguments) would only
use
a minimal set of structure.member offsets.
Thanks,
Dave
4:09 p.m.
----- Original Message -----
> > My suggestion re: a new -g flag would have it applicable to
both CFS and RT
> > run queues (even though the RT groups are currently being shown). I was
> > thinking that it might be safest to have "runq" (with no argument) to
just
> > display a simple list of tasks, and only show group information with -g.
> > Are you suggesting that -g would only apply to throttled cfs_rqs/rt_rqs?
> >
>
> nop, '-g' is ok to show all rqs with group information.
>
> I rewrote the patch by adding option '-g' to runq and not changing any
behaviour of
> the runq (with no argument).
>
> TODO:
> 1. The help info about the -g option.
>
> I still have a suggestion: when we dump cfs_rq, we don't dump it
hierarchically,
> but for the current rt_rq, we dump it hierarchically with group info.
> So could I make another patch to remove the rt_rq hierarchy, just make it
> displayed identical with cfs_rq display.(For -g will display group info for both
rt_rq
> and cfs_rq)
Yes, that's exactly what I was suggesting.
I worry about the large number of kernel structure.member declarations that the
command depends upon, because if just one of them changes, it breaks the command.
So it would be preferable if the "runq" command (with no arguments) would only
use
a minimal set of structure.member offsets.
Thanks,
Dave
And to follow up, I'm still running tests (and will do so overnight) on your latest
patch, but I immediately seen this on any 2.6.30, 2.6.31 or 2.6.32 kernel, and on
some 2.6.36 and 2.6.38 kernels, where "runq -g" fails like this:
crash> runq -g
runq: invalid kernel virtual address: 0 type: "dentry"
crash>
And we certainly want to keep the group information separate from
the normal "runq" command. Here on a "live" 4 cpu Fedora 3.6.3
kernel,
the command output exceeds 1000 lines! I'm pretty sure that most
people will *not* want to see all of this:
crash> runq -g
CPU 0
CURRENT: PID: 0 TASK: ffffffff81c13420 COMMAND: "swapper/0"
RT PRIO_ARRAY: ffff88021e213e28
[100] GROUP RT PRIO_ARRAY: ffff88020db22000 <system>
[100] GROUP RT PRIO_ARRAY: ffff8801f8180000 <udisks2.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8e11800 <colord-sane.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8802116f0000 <colord.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f1fd000 <cups.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f51800 <upower.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f994c800 <accounts-daemon.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880212ae1000 <prefdm.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8180800 <atd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9abe000 <crond.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9aba000 <autofs.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f54000 <sm-client.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f7ee6000 <sendmail.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f7e21800 <nscd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f7d00800 <wpa_supplicant.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f6a000 <ypbind.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f6d800 <nfs-lock.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8e12800 <rpcbind.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8d7a800 <libvirtd.service>
[100] GROUP RT PRIO_ARRAY: ffff8801f9944000 <libvirt>
[100] GROUP RT PRIO_ARRAY: ffff8801f7e22800 <lxc>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88021091b000 <qemu>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211f4e000 <sshd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211f4a800 <ksmtuned.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9a7c000 <dbus.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9a78800 <rsyslog.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f99cd800 <avahi-daemon.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f99c9800
<system-setup-keyboard.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9975800 <systemd-logind.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801fc2cb800 <irqbalance.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f98c6800 <abrt-xorg.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9885800 <NetworkManager.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020ee39800 <acpid.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020ee3d800 <gpm.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f40d800 <mcelog.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211ed9800 <abrt-oops.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f403000 <abrtd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f400800 <smartd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020dc47800 <auditd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211a55000 <iscsid.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8802108ab800 <bluetooth.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211f4a000 <fsck@.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211edc000 <udev.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880212adc000 <systemd-journald.service>
[no tasks queued]
CFS RB_ROOT: ffff880215806068
GROUP CFS RB_ROOT: ffff8801ed860728
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ed861a28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dab6228
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd98e28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ef1ec828
[no tasks queued]
GROUP CFS RB_ROOT: ffff8800a07d6d28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ef2ad928
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f11fe228
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f180b628
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ee24528
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e110828
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210a5fe28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e161c28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f503b28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f5b128
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ff7c28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d972828
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210a5f328
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e115c28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f5d128
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802129c2b28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d6a028
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f7d3e328
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f822ed28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8fbcd28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8fbde28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f822ec28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d77d28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd59328
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8e7d528
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8fbdb28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f59428
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8e7e128
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ee78b28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d983f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ed4028
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9b92428
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9b03128
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020edd4928
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a71628
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a70028
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcfe128
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ec9b28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020edd5328
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d924028
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e69228
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd59428
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211f26f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e68528
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f75e828
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f75e528
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ef0e28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d972f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e2aac28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e110428
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020db0cc28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ee1428 <system>
GROUP CFS RB_ROOT: ffff88020ed3f928 <udisks2.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f814c028 <colord-sane.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd0eb28 <colord.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88021041b328 <cups.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd98228 <upower.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210aab628 <accounts-daemon.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f7f6d728 <prefdm.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f7f6c028 <atd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d01e28 <crond.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d01728 <autofs.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d980228 <sm-client.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f5dd28 <sendmail.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d77c28 <nscd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e152f28 <wpa_supplicant.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d6bb28 <ypbind.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d943b28 <nfs-lock.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ec8428 <rpcbind.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d981728 <libvirtd.service>
GROUP CFS RB_ROOT: ffff8801f82b6828 <libvirt>
GROUP CFS RB_ROOT: ffff8801f828d828 <lxc>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f82b7628 <qemu>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801fc232d28 <sshd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a70428 <ksmtuned.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ddf7128 <dbus.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a77728 <rsyslog.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd53e28 <avahi-daemon.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcf6728 <system-setup-keyboard.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e68c28 <systemd-logind.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e3f928 <irqbalance.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f638828 <abrt-xorg.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210800c28 <NetworkManager.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcfe928 <acpid.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d982628 <gpm.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ff68d28 <mcelog.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d982428 <abrt-oops.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d983928 <abrtd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211a5b728 <smartd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210af7528 <auditd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802116f9c28 <iscsid.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e2aa328 <bluetooth.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d942828 <fsck@.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211a5a128 <udev.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802116f8328 <systemd-journald.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020a844628
[no tasks queued]
CPU 1
CURRENT: PID: 1122 TASK: ffff8801f8e18000 COMMAND: "Xorg"
RT PRIO_ARRAY: ffff88021e253e28
[100] GROUP RT PRIO_ARRAY: ffff88020db21800 <system>
[100] GROUP RT PRIO_ARRAY: ffff8801f8187800 <udisks2.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8e10800 <colord-sane.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8802116f5800 <colord.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f1fc800 <cups.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f50000 <upower.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f994c000 <accounts-daemon.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880212ae3000 <prefdm.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8181000 <atd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9abe800 <crond.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9aba800 <autofs.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f52000 <sm-client.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f7ee5800 <sendmail.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f7e20800 <nscd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f7d01000 <wpa_supplicant.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f69800 <ypbind.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f6e000 <nfs-lock.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8e13000 <rpcbind.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8d7b000 <libvirtd.service>
[100] GROUP RT PRIO_ARRAY: ffff8801f9943000 <libvirt>
[100] GROUP RT PRIO_ARRAY: ffff8801f7e23000 <lxc>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88021091a800 <qemu>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880212bdf800 <sshd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211f4b000 <ksmtuned.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9a7c800 <dbus.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9a78000 <rsyslog.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f99ce000 <avahi-daemon.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f99ca000
<system-setup-keyboard.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9976000 <systemd-logind.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801fc2cd000 <irqbalance.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f98c7000 <abrt-xorg.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9886000 <NetworkManager.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020ee39000 <acpid.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020ee3d000 <gpm.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f40d000 <mcelog.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211ed9000 <abrt-oops.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f401000 <abrtd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f404800 <smartd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801fc2c8000 <auditd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211a53000 <iscsid.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8802108ac800 <bluetooth.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211f4d800 <fsck@.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211edc800 <udev.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880212adc800 <systemd-journald.service>
[no tasks queued]
CFS RB_ROOT: ffff880215806068
GROUP CFS RB_ROOT: ffff880211ee1b28 <system>
GROUP CFS RB_ROOT: ffff8801f7f6d528 <prefdm.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ed3fa28 <udisks2.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f814dc28 <colord-sane.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd0ff28 <colord.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88021041a928 <cups.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd98128 <upower.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210aab928 <accounts-daemon.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f7f6d228 <atd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d01228 <crond.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d00c28 <autofs.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d980528 <sm-client.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f5c228 <sendmail.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d76b28 <nscd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e153028 <wpa_supplicant.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d6bc28 <ypbind.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d942d28 <nfs-lock.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801fc177928 <rpcbind.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d981d28 <libvirtd.service>
GROUP CFS RB_ROOT: ffff8801f82b6128 <libvirt>
GROUP CFS RB_ROOT: ffff8801f828cb28 <lxc>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f82b7228 <qemu>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801fc232f28 <sshd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a70928 <ksmtuned.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ddf7828 <dbus.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a77828 <rsyslog.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd52a28 <avahi-daemon.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcf7428 <system-setup-keyboard.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e68328 <systemd-logind.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e3eb28 <irqbalance.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f639b28 <abrt-xorg.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210800b28 <NetworkManager.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcff828 <acpid.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d983828 <gpm.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ff69d28 <mcelog.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d983728 <abrt-oops.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d983d28 <abrtd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211a5ab28 <smartd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210af7228 <auditd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802116f8e28 <iscsid.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e2aa928 <bluetooth.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d943828 <fsck@.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e95ba28 <udev.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802116f8228 <systemd-journald.service>
[no tasks queued]
[120] PID: 27130 TASK: ffff8801c2c3ae20 COMMAND: "kworker/1:2"
GROUP CFS RB_ROOT: ffff8801ed860028
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ed860828
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dab6e28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ef2d4e28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ef1ed928
[no tasks queued]
GROUP CFS RB_ROOT: ffff8800a07d7f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ef2aca28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f11ff228
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f180b828
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ee24f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e111428
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210a5fd28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e160028
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f502528
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f5b528
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ff7d28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ddf7728
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210a5e828
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e114e28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f5c028
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802129c2c28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d6a828
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f7d3f028
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f822f028
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8fbce28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8fbd028
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f822eb28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d76e28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f58628
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8e7c728
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8fbd828
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f59528
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8e7e228
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ee78928
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d983028
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ed5d28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9b93928
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9b03f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020edd5f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a71e28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e2aad28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcfe328
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ec8728
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020edd4628
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d925328
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e69628
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd58028
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211f26a28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e68f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ef0c28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ef1328
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ef1028
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d972728
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e2aaa28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020db0c228
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020db0c728
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020a844328
[no tasks queued]
CPU 2
CURRENT: PID: 0 TASK: ffff880212958000 COMMAND: "swapper/2"
RT PRIO_ARRAY: ffff88021e293e28
[100] GROUP RT PRIO_ARRAY: ffff88020db22800 <system>
[100] GROUP RT PRIO_ARRAY: ffff8801f7ee7000 <udisks2.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8e10000 <colord-sane.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8802116f7000 <colord.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f1fb800 <cups.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f53800 <upower.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f994b800 <accounts-daemon.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880212ae4000 <prefdm.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8181800 <atd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9abf000 <crond.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9abb000 <autofs.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f56000 <sm-client.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f7ee5000 <sendmail.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f7e21000 <nscd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f7d01800 <wpa_supplicant.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f69000 <ypbind.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f6e800 <nfs-lock.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8e13800 <rpcbind.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8d7b800 <libvirtd.service>
[100] GROUP RT PRIO_ARRAY: ffff8801f8079000 <libvirt>
[100] GROUP RT PRIO_ARRAY: ffff8801f7e23800 <lxc>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88021091a000 <qemu>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020dc47000 <sshd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211f49800 <ksmtuned.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9a7d000 <dbus.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9a79000 <rsyslog.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f99ce800 <avahi-daemon.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f99ca800
<system-setup-keyboard.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9976800 <systemd-logind.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801fc2cd800 <irqbalance.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f98c7800 <abrt-xorg.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9886800 <NetworkManager.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020ee38800 <acpid.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020ee3c800 <gpm.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f40c800 <mcelog.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211ed8800 <abrt-oops.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f401800 <abrtd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f405000 <smartd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801fc2c8800 <auditd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211a55800 <iscsid.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020eee2000 <bluetooth.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020e634000 <fsck@.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211edd000 <udev.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880212add000 <systemd-journald.service>
[no tasks queued]
CFS RB_ROOT: ffff880215806068
GROUP CFS RB_ROOT: ffff8801ed861928
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ed861c28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dab6928
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ef2d5028
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ef1edf28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8800a07d6428
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ef2ada28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f11fe428
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f180b128
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ee25728
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e110728
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210a5f928
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e160b28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e114428
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f5b028
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ff7e28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f822ff28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210a5f628
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e115f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f5c828
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802129c2028
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d6a928
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f7d3f928
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f822f428
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8fbcf28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8fbd228
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f822e928
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d76328
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f58428
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8e7ca28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8fbc128
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f59628
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8e7e328
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a73228
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801fc3e8628
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ed5c28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9b93128
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9b03b28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211a5a528
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a71f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f02e528
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcfeb28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ec9428
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020edd5828
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d924c28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e68228
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d9f3d28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211f27928
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e68928
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e111e28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ef0f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ef1d28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d973228
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e2ab128
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020db0d028
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020db0d628
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ee1628 <system>
GROUP CFS RB_ROOT: ffff88020ed3fb28 <udisks2.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f814cb28 <colord-sane.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd0fc28 <colord.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88021041a228 <cups.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd99628 <upower.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210aaba28 <accounts-daemon.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f7f6df28 <prefdm.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f7f6d428 <atd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d00628 <crond.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d00428 <autofs.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d981b28 <sm-client.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f5c328 <sendmail.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d77a28 <nscd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8e99428 <wpa_supplicant.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d6bd28 <ypbind.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d943528 <nfs-lock.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801fc232328 <rpcbind.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d980b28 <libvirtd.service>
GROUP CFS RB_ROOT: ffff8801f82b6328 <libvirt>
GROUP CFS RB_ROOT: ffff8801f828de28 <lxc>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f82b6928 <qemu>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801fc233128 <sshd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a70728 <ksmtuned.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ddf6528 <dbus.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a77928 <rsyslog.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd53028 <avahi-daemon.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcf6e28 <system-setup-keyboard.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e3f528 <systemd-logind.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e3e428 <irqbalance.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f638128 <abrt-xorg.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210800028 <NetworkManager.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcfec28 <acpid.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d982828 <gpm.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ff69028 <mcelog.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d982c28 <abrt-oops.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d982728 <abrtd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211a5a228 <smartd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802116f8f28 <auditd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802116f9728 <iscsid.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e2ab328 <bluetooth.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d942528 <fsck@.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802116f8a28 <udev.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802116f8528 <systemd-journald.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020a844f28
[no tasks queued]
CPU 3
CURRENT: PID: 4733 TASK: ffff8801eed74530 COMMAND: "crash"
RT PRIO_ARRAY: ffff88021e2d3e28
[100] GROUP RT PRIO_ARRAY: ffff88020db23000 <system>
[100] GROUP RT PRIO_ARRAY: ffff8801f7ee0000 <udisks2.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8e17000 <colord-sane.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8802116f1000 <colord.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f1fa000 <cups.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f53000 <upower.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f994f800 <accounts-daemon.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880212ae4800 <prefdm.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8182000 <atd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9abf800 <crond.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9abb800 <autofs.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f56800 <sm-client.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f7ee4800 <sendmail.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f7e20000 <nscd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f7d02000 <wpa_supplicant.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f68800 <ypbind.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8f6f000 <nfs-lock.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8e14000 <rpcbind.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f8d7c000 <libvirtd.service>
[100] GROUP RT PRIO_ARRAY: ffff880211ec1000 <libvirt>
[100] GROUP RT PRIO_ARRAY: ffff8801f7e24000 <lxc>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88021091c800 <qemu>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8802108ab000 <sshd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211f4e800 <ksmtuned.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9a7d800 <dbus.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9a79800 <rsyslog.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f99cf000 <avahi-daemon.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f99cb000
<system-setup-keyboard.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9977000 <systemd-logind.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801fc2ce000 <irqbalance.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9948000 <abrt-xorg.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801f9887000 <NetworkManager.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020ee38000 <acpid.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020ee3c000 <gpm.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f408000 <mcelog.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211ed8000 <abrt-oops.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f403800 <abrtd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020f405800 <smartd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff8801fc2c9000 <auditd.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211a53800 <iscsid.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020eee1000 <bluetooth.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff88020e632800 <fsck@.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880211edd800 <udev.service>
[no tasks queued]
[100] GROUP RT PRIO_ARRAY: ffff880212add800 <systemd-journald.service>
[no tasks queued]
CFS RB_ROOT: ffff880215806068
GROUP CFS RB_ROOT: ffff8801ed860b28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ef2b7b28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcff028
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ef2d4528
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ef1ecf28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8800a07d7b28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801ef2ad528
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f11ff828
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f180a128
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ee24928
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e111028
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210a5fa28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e161a28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a73928
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f5ae28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ff7f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f822fe28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210a5e428
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e115b28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f5d228
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802129c2228
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d6b128
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f7d3ed28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f822f628
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8fbc028
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8fbd128
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f822e828
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d76228
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8e7d228
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8e7cb28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8fbc428
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f59728
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8e7e428
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020edd4a28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801fc3e8b28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ed5628
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9b92f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9b03e28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211a5bb28
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a71528
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f02ef28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcfea28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ec9728
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020edd5e28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d925e28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e69f28
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ed5928
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211f26528
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e69d28
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e111928
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ef1628
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f75e728
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d972528
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e2aa028
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020db0d528
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020db0c928
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211ee1f28 <system>
GROUP CFS RB_ROOT: ffff88020ed3fc28 <udisks2.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f52de28 <colord-sane.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd0fe28 <colord.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f006128 <cups.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd99728 <upower.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210aab228 <accounts-daemon.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f7f6c328 <prefdm.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f7f6d028 <atd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d00228 <crond.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d01828 <autofs.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f638c28 <sm-client.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8f5cc28 <sendmail.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d76928 <nscd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8e99128 <wpa_supplicant.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f8d6be28 <ypbind.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e152b28 <nfs-lock.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801fc233b28 <rpcbind.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d980428 <libvirtd.service>
GROUP CFS RB_ROOT: ffff8801f82b6428 <libvirt>
GROUP CFS RB_ROOT: ffff8801f828c928 <lxc>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f82b6b28 <qemu>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801fc233328 <sshd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a70c28 <ksmtuned.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ddf7a28 <dbus.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8801f9a77a28 <rsyslog.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dd53b28 <avahi-daemon.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcf7628 <system-setup-keyboard.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e3fb28 <systemd-logind.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211e3f428 <irqbalance.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020f638528 <abrt-xorg.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210801428 <NetworkManager.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020dcff128 <acpid.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d983e28 <gpm.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020ff68628 <mcelog.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d982a28 <abrt-oops.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d982128 <abrtd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880211a5a028 <smartd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020de77828 <auditd.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff880210af7a28 <iscsid.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020e2ab828 <bluetooth.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020d943d28 <fsck@.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802116f9128 <udev.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff8802116f8828 <systemd-journald.service>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020a844128
[no tasks queued]
crash>
In fact, it's difficult to actually find a real *task* that is on
a run queue from among all of the empty "[no tasks queued]" groups!
Dave
9:27 a.m.
----- Original Message -----
And to follow up, I'm still running tests (and will do so
overnight) on your latest
patch, but I immediately see this on any 2.6.30, 2.6.31 or 2.6.32 kernel, and on
some 2.6.36 and 2.6.38 kernels, where "runq -g" fails like this:
crash> runq -g
runq: invalid kernel virtual address: 0 type: "dentry"
crash>
The error above is pretty persistent on all kernels from 2.6.27 to 2.6.38,
where you can get a legitimate cgroup that has no dentry:
+static void
+get_task_group_name(ulong group, char **group_name)
+{
+ ulong cgroup, dentry, name;
+ char *dentry_buf, *tmp;
+ int len;
+
+ readmem(group + OFFSET(task_group_css) + OFFSET(cgroup_subsys_state_cgroup),
+ KVADDR, &cgroup, sizeof(ulong),
+ "task_group css cgroup", FAULT_ON_ERROR);
+ if (cgroup == 0)
+ return;
+
+ readmem(cgroup + OFFSET(cgroup_dentry), KVADDR, &dentry, sizeof(ulong),
+ "cgroup dentry", FAULT_ON_ERROR);
+
+ dentry_buf = GETBUF(SIZE(dentry));
+ readmem(dentry, KVADDR, dentry_buf, SIZE(dentry),
+ "dentry", FAULT_ON_ERROR);
Dave
8:52 p.m.
于 2012年11月02日 05:09, Dave Anderson 写道:
----- Original Message -----
>>> My suggestion re: a new -g flag would have it applicable to both CFS and RT
>>> run queues (even though the RT groups are currently being shown). I was
>>> thinking that it might be safest to have "runq" (with no argument)
to just
>>> display a simple list of tasks, and only show group information with -g.
>>> Are you suggesting that -g would only apply to throttled cfs_rqs/rt_rqs?
>>>
>>
>> nop, '-g' is ok to show all rqs with group information.
>>
>> I rewrote the patch by adding option '-g' to runq and not changing any
behaviour of
>> the runq (with no argument).
>>
>> TODO:
>> 1. The help info about the -g option.
>>
>> I still have a suggestion: when we dump cfs_rq, we don't dump it
hierarchically,
>> but for the current rt_rq, we dump it hierarchically with group info.
>> So could I make another patch to remove the rt_rq hierarchy, just make it
>> displayed identical with cfs_rq display.(For -g will display group info for both
rt_rq
>> and cfs_rq)
>
> Yes, that's exactly what I was suggesting.
>
> I worry about the large number of kernel structure.member declarations that the
> command depends upon, because if just one of them changes, it breaks the command.
> So it would be preferable if the "runq" command (with no arguments) would
only use
> a minimal set of structure.member offsets.
I made two new offset-init functions for runq:
static void cfs_rq_offset_init(void);
static void task_group_offset_init(void);
the first one is used for runq with no arguments. The extra offset init for runq -g
is in the second one. So nothing changed with the original runq after applying the
runq -g patch.
>
> Thanks,
> Dave
And to follow up, I'm still running tests (and will do so overnight) on your latest
patch, but I immediately seen this on any 2.6.30, 2.6.31 or 2.6.32 kernel, and on
some 2.6.36 and 2.6.38 kernels, where "runq -g" fails like this:
crash> runq -g
runq: invalid kernel virtual address: 0 type: "dentry"
crash>
As you have pointed in another mail, this is fixed in patch v2, I think.
And we certainly want to keep the group information separate from
the normal "runq" command. Here on a "live" 4 cpu Fedora 3.6.3
kernel,
the command output exceeds 1000 lines! I'm pretty sure that most
people will *not* want to see all of this:
crash> runq -g
CPU 0
CURRENT: PID: 0 TASK: ffffffff81c13420 COMMAND: "swapper/0"
RT PRIO_ARRAY: ffff88021e213e28
[100] GROUP RT PRIO_ARRAY: ffff88020db22000 <system>
[100] GROUP RT PRIO_ARRAY: ffff8801f8180000 <udisks2.service>
[no tasks queued]
<snip>
[no tasks queued]
GROUP CFS RB_ROOT: ffff88020a844128
[no tasks queued]
crash>
In fact, it's difficult to actually find a real *task* that is on
a run queue from among all of the empty "[no tasks queued]" groups!
I've noticed this in the first patch and patch V2 has fixed this.
So I attach the new patch v2 version for runq -g. If you still find
any bug in your tests or have any suggestion about it, that's very
helpful.
TODO:
1. The help info about the -g option.
2. Change rt_rq tasks displayed non-hierarchically.
Many thanks
Zhang
3:44 p.m.
----- Original Message -----
>>
>> I worry about the large number of kernel structure.member declarations that the
>> command depends upon, because if just one of them changes, it breaks the
command.
>> So it would be preferable if the "runq" command (with no arguments)
would only use
>> a minimal set of structure.member offsets.
I made two new offset-init functions for runq:
static void cfs_rq_offset_init(void);
static void task_group_offset_init(void);
One minor suggestion -- it's really not necessary to call MEMBER_EXISTS()
prior to calling MEMBER_OFFSET_INIT():
diff --git a/kernel.c b/kernel.c
index 45da48e..868d777 100755
--- a/kernel.c
+++ b/kernel.c
@@ -308,6 +308,12 @@ kernel_init()
STRUCT_SIZE_INIT(prio_array, "prio_array");
MEMBER_OFFSET_INIT(rq_cfs, "rq", "cfs");
+ if (MEMBER_EXISTS("task_group", "cfs_rq"))
+ MEMBER_OFFSET_INIT(task_group_cfs_rq, "task_group",
"cfs_rq");
+ if (MEMBER_EXISTS("task_group", "rt_rq"))
+ MEMBER_OFFSET_INIT(task_group_rt_rq, "task_group",
"rt_rq");
+ if (MEMBER_EXISTS("task_group", "parent"))
+ MEMBER_OFFSET_INIT(task_group_parent, "task_group",
"parent");
You can simply just call MEMBER_OFFSET_INIT() the 3 times above. If the structure
members don't exist, MEMBER_OFFSET_INIT() will just set the offsets to -1
(INVALID_OFFSET).
The same thing applies in your task_group_offset_init() function:
+ if (MEMBER_EXISTS("task_group", "cfs_bandwidth")) {
+ MEMBER_OFFSET_INIT(task_group_cfs_bandwidth,
+ "task_group", "cfs_bandwidth");
+ MEMBER_OFFSET_INIT(cfs_rq_throttled, "cfs_rq",
+ "throttled");
+ }
+
+ if (MEMBER_EXISTS("task_group", "rt_bandwidth")) {
+ MEMBER_OFFSET_INIT(task_group_rt_bandwidth,
+ "task_group", "rt_bandwidth");
+ MEMBER_OFFSET_INIT(rt_rq_rt_throttled, "rt_rq",
+ "rt_throttled");
+ }
Sometimes it helps to call MEMBER_EXISTS() for clarity's sake, but it's
actually kind of redundant.
>
> And to follow up, I'm still running tests (and will do so overnight) on your
latest
> patch, but I immediately see this on any 2.6.30, 2.6.31 or 2.6.32 kernel, and on
> some 2.6.36 and 2.6.38 kernels, where "runq -g" fails like this:
>
> crash> runq -g
> runq: invalid kernel virtual address: 0 type: "dentry"
> crash>
>
As you have pointed in another mail, this is fixed in patch v2, I think.
Yes, that's fixed this time...
> And we certainly want to keep the group information separate from
> the normal "runq" command. Here on a "live" 4 cpu Fedora 3.6.3
kernel,
> the command output exceeds 1000 lines! I'm pretty sure that most
> people will *not* want to see all of this:
>
> crash> runq -g
> CPU 0
> CURRENT: PID: 0 TASK: ffffffff81c13420 COMMAND: "swapper/0"
> RT PRIO_ARRAY: ffff88021e213e28
> [100] GROUP RT PRIO_ARRAY: ffff88020db22000 <system>
> [100] GROUP RT PRIO_ARRAY: ffff8801f8180000 <udisks2.service>
> [no tasks queued]
<snip>
> [no tasks queued]
> GROUP CFS RB_ROOT: ffff88020a844128
> [no tasks queued]
> crash>
>
> In fact, it's difficult to actually find a real *task* that is on
> a run queue from among all of the empty "[no tasks queued]" groups!
>
I've noticed this in the first patch and patch V2 has fixed this.
OK, although I didn't realize that was a bug? What were all of those
empty groups that are no longer shown? Are they actually *not* queued?
So I attach the new patch v2 version for runq -g. If you still find
any bug in your tests or have any suggestion about it, that's very
helpful.
TODO:
1. The help info about the -g option.
2. Change rt_rq tasks displayed non-hierarchically.
Like I mentioned above, the latest patch does not change the default
behavior of runq alone, and "runq -g" is not as verbose as the last
patch, which I presume is your intent.
Thanks,
Dave
2:58 a.m.
于 2012年11月06日 05:44, Dave Anderson 写道:
----- Original Message -----
>>>
>>> I worry about the large number of kernel structure.member declarations that
the
>>> command depends upon, because if just one of them changes, it breaks the
command.
>>> So it would be preferable if the "runq" command (with no arguments)
would only use
>>> a minimal set of structure.member offsets.
>
> I made two new offset-init functions for runq:
>
> static void cfs_rq_offset_init(void);
> static void task_group_offset_init(void);
One minor suggestion -- it's really not necessary to call MEMBER_EXISTS()
prior to calling MEMBER_OFFSET_INIT():
diff --git a/kernel.c b/kernel.c
index 45da48e..868d777 100755
--- a/kernel.c
+++ b/kernel.c
@@ -308,6 +308,12 @@ kernel_init()
STRUCT_SIZE_INIT(prio_array, "prio_array");
MEMBER_OFFSET_INIT(rq_cfs, "rq", "cfs");
+ if (MEMBER_EXISTS("task_group", "cfs_rq"))
+ MEMBER_OFFSET_INIT(task_group_cfs_rq, "task_group",
"cfs_rq");
+ if (MEMBER_EXISTS("task_group", "rt_rq"))
+ MEMBER_OFFSET_INIT(task_group_rt_rq, "task_group",
"rt_rq");
+ if (MEMBER_EXISTS("task_group", "parent"))
+ MEMBER_OFFSET_INIT(task_group_parent, "task_group",
"parent");
You can simply just call MEMBER_OFFSET_INIT() the 3 times above. If the structure
members don't exist, MEMBER_OFFSET_INIT() will just set the offsets to -1
(INVALID_OFFSET).
The same thing applies in your task_group_offset_init() function:
+ if (MEMBER_EXISTS("task_group", "cfs_bandwidth")) {
+ MEMBER_OFFSET_INIT(task_group_cfs_bandwidth,
+ "task_group", "cfs_bandwidth");
+ MEMBER_OFFSET_INIT(cfs_rq_throttled, "cfs_rq",
+ "throttled");
+ }
+
+ if (MEMBER_EXISTS("task_group", "rt_bandwidth")) {
+ MEMBER_OFFSET_INIT(task_group_rt_bandwidth,
+ "task_group", "rt_bandwidth");
+ MEMBER_OFFSET_INIT(rt_rq_rt_throttled, "rt_rq",
+ "rt_throttled");
+ }
Sometimes it helps to call MEMBER_EXISTS() for clarity's sake, but it's
actually kind of redundant.
Thanks. Fixed.
>>
>> And to follow up, I'm still running tests (and will do so overnight) on your
latest
>> patch, but I immediately see this on any 2.6.30, 2.6.31 or 2.6.32 kernel, and on
>> some 2.6.36 and 2.6.38 kernels, where "runq -g" fails like this:
>>
>> crash> runq -g
>> runq: invalid kernel virtual address: 0 type: "dentry"
>> crash>
>>
>
> As you have pointed in another mail, this is fixed in patch v2, I think.
Yes, that's fixed this time...
>
>> And we certainly want to keep the group information separate from
>> the normal "runq" command. Here on a "live" 4 cpu Fedora
3.6.3 kernel,
>> the command output exceeds 1000 lines! I'm pretty sure that most
>> people will *not* want to see all of this:
>>
>> crash> runq -g
>> CPU 0
>> CURRENT: PID: 0 TASK: ffffffff81c13420 COMMAND: "swapper/0"
>> RT PRIO_ARRAY: ffff88021e213e28
>> [100] GROUP RT PRIO_ARRAY: ffff88020db22000 <system>
>> [100] GROUP RT PRIO_ARRAY: ffff8801f8180000 <udisks2.service>
>> [no tasks queued]
>
> <snip>
>
>> [no tasks queued]
>> GROUP CFS RB_ROOT: ffff88020a844128
>> [no tasks queued]
>> crash>
>>
>> In fact, it's difficult to actually find a real *task* that is on
>> a run queue from among all of the empty "[no tasks queued]" groups!
>>
>
> I've noticed this in the first patch and patch V2 has fixed this.
OK, although I didn't realize that was a bug? What were all of those
empty groups that are no longer shown? Are they actually *not* queued?
Yes, cfs_rq and rt_rq owned by empty groups actually not queued.
every group has its own cfs_rq and rt_rq, if there is no task in this group
and no task in any of its child groups, it is an empty group and its cfs_rq
and rt_rq will not be queued in the parent runqueue.
actually, throttled cfs_rq(rt_rq) is also dequeued from its parent cfs_rq(rt_rq),
but we should display them for there are task(s) in it. for example:
1 CPU 0
2 CURRENT: PID: 14734 TASK: ffff88010626f500 COMMAND: "sh"
3 RT PRIO_ARRAY: ffff880028216808
4 [ 0] GROUP RT PRIO_ARRAY: ffff880139fc9800 <test1> (THROTTLED)
5 [ 0] PID: 14750 TASK: ffff88013a4dd540 COMMAND: "rtloop99"
6 [ 1] PID: 14748 TASK: ffff88013bbca040 COMMAND: "rtloop98"
7 [ 1] GROUP RT PRIO_ARRAY: ffff880089029000 <test11>
8 [ 1] PID: 14752 TASK: ffff880088abf500 COMMAND:
"rtloop98"
9 [ 54] PID: 14749 TASK: ffff880037a4e080 COMMAND: "rtloop45"
10 [ 98] PID: 14746 TASK: ffff88012678c080 COMMAND: "rtloop1"
11 CFS RB_ROOT: ffff88013fc23050
12 [120] PID: 14740 TASK: ffff88013b1e6080 COMMAND: "sh"
13 [120] PID: 14738 TASK: ffff88012678d540 COMMAND: "sh"
14 GROUP CFS RB_ROOT: ffff8800897af430 <test2> (THROTTLED)
15 [120] PID: 14732 TASK: ffff88013bbcb500 COMMAND: "sh"
16 [120] PID: 14728 TASK: ffff8800b3496080 COMMAND: "sh"
17 [120] PID: 14730 TASK: ffff880037833540 COMMAND: "sh"
18 GROUP CFS RB_ROOT: ffff880037943e30 <test1> (THROTTLED)
19 [120] PID: 14726 TASK: ffff880138d42aa0 COMMAND: "sh"
>
> So I attach the new patch v2 version for runq -g. If you still find
> any bug in your tests or have any suggestion about it, that's very
> helpful.
>
> TODO:
> 1. The help info about the -g option.
> 2. Change rt_rq tasks displayed non-hierarchically.
Like I mentioned above, the latest patch does not change the default
behavior of runq alone, and "runq -g" is not as verbose as the last
patch, which I presume is your intent.
Two patches added. One is the fixed version for v2 and the other is
adding the help info for run -g.
Thanks
Zhang
3:30 p.m.
----- Original Message -----
>
>>
>> So I attach the new patch v2 version for runq -g. If you still find
>> any bug in your tests or have any suggestion about it, that's very
>> helpful.
>>
>> TODO:
>> 1. The help info about the -g option.
>> 2. Change rt_rq tasks displayed non-hierarchically.
>
> Like I mentioned above, the latest patch does not change the default
> behavior of runq alone, and "runq -g" is not as verbose as the last
> patch, which I presume is your intent.
Two patches added. One is the fixed version for v2 and the other is
adding the help info for run -g.
Thanks
Zhang
These latest patches tested OK.
However, I must admit that I don't really understand the details w/respect
to the difference between the "runq" and "runq -g" output other than
the
two different options essentially find all per-cpu queued tasks coming from
different rb_root structures. Also, the "runq -g" implementation is remarkably
complex in comparison to the "runq" alone, making its future maintenance a
potentially difficult chore because of the huge number of structure.member
dependencies.
Now, for the most part, you have been able to segregate the code, *except*
for the overloading of the currently-existing dump_tasks_in_cfs_rq() function to
handle both the "runq" and the "runq -g" commands, i.e., where
you've merged in
the g_flag and depth business. And as a result, it makes the patched
"dual-mode"
dump_tasks_in_cfs_rq() function difficult to understand.
Here is my suggestion/request:
(1) please leave dump_CFS_runqueues() and dump_tasks_in_cfs_rq() as they are now.
(2) create a new (somewhat redundant) "dump_tasks_in_task_group_rq()" function
that is only
used for "runq -g".
(3) have your new dump_tasks_by_task_group() function call the new
dump_tasks_in_task_group_rq()
function.
That would completely segregate the implementations of the two options. The worst
case for doing that it that way is that there will be dump_tasks_in_cfs_rq() and
and dump_tasks_in_task_group_rq() functions that are somewhat similar. But considering
the huge amount of code that you are adding for "runq -g", having separate
functions is
hardly a cause for concern. And it should be fairly trivial for you to update your patch
to do it that way.
And then you could effectively do the same type of code separation for the RT queues.
Does that make sense to you?
Dave
4:26 a.m.
于 2012年11月07日 05:30, Dave Anderson 写道:
----- Original Message -----
>>
>>>
>>> So I attach the new patch v2 version for runq -g. If you still find
>>> any bug in your tests or have any suggestion about it, that's very
>>> helpful.
>>>
>>> TODO:
>>> 1. The help info about the -g option.
>>> 2. Change rt_rq tasks displayed non-hierarchically.
>>
>> Like I mentioned above, the latest patch does not change the default
>> behavior of runq alone, and "runq -g" is not as verbose as the last
>> patch, which I presume is your intent.
>
>
> Two patches added. One is the fixed version for v2 and the other is
> adding the help info for run -g.
>
> Thanks
> Zhang
These latest patches tested OK.
However, I must admit that I don't really understand the details w/respect
to the difference between the "runq" and "runq -g" output other than
the
two different options essentially find all per-cpu queued tasks coming from
different rb_root structures. Also, the "runq -g" implementation is
remarkably
complex in comparison to the "runq" alone, making its future maintenance a
potentially difficult chore because of the huge number of structure.member
dependencies.
Hmm, runq -g includes tasks in throttled cfs_rqs/rt_rqs. Its implementation is
so complex because when dumping a cfs rb_tree, if one of its node is a also child
cfs rb_tree, we follow the child tree and dump all tasks in it. If the child cfs
rb_tree is throttled, it is dequeued from its parent rb_tree and we lost tasks in
this rb_tree. So when dumping the cfs rb_tree, I checked if there is a child cfs
rb_tree throttled, if so, dumping it.
The same thing is for rt_rq.
Dumping group info, such as group name, is to make it clear that which task is
belong to which group.
Now, for the most part, you have been able to segregate the code, *except*
for the overloading of the currently-existing dump_tasks_in_cfs_rq() function to
handle both the "runq" and the "runq -g" commands, i.e., where
you've merged in
the g_flag and depth business. And as a result, it makes the patched
"dual-mode"
dump_tasks_in_cfs_rq() function difficult to understand.
Here is my suggestion/request:
(1) please leave dump_CFS_runqueues() and dump_tasks_in_cfs_rq() as they are now.
(2) create a new (somewhat redundant) "dump_tasks_in_task_group_rq()" function
that is only
used for "runq -g".
(3) have your new dump_tasks_by_task_group() function call the new
dump_tasks_in_task_group_rq()
function.
That would completely segregate the implementations of the two options. The worst
case for doing that it that way is that there will be dump_tasks_in_cfs_rq() and
and dump_tasks_in_task_group_rq() functions that are somewhat similar. But considering
the huge amount of code that you are adding for "runq -g", having separate
functions is
hardly a cause for concern. And it should be fairly trivial for you to update your
patch
to do it that way.
And then you could effectively do the same type of code separation for the RT queues.
Does that make sense to you?
ok. I rewrite the patch and they are tested ok in my box.
Thanks
Zhang
1:15 p.m.
----- Original Message -----
ok. I rewrite the patch and they are tested ok in my box.
Thanks
Zhang
My tests weren't so successful this time, and I also have some questions
about the runq -g output.
I tested your latest patches on a sample set of 70 dumpfiles whose
kernels all use CFS runqueues. In 7 of the 70 "runq -g" tests,
the command caused the crash session to fail like so:
crash> runq -g
CPU 0
CURRENT: PID: 0 TASK: ffffffff81c0d020 COMMAND: "swapper/0"
RT PRIO_ARRAY: ffff8802277d4590
[no tasks queued]
CFS RB_ROOT: ffff880227018070
[120] PID: 3 TASK: ffff8802221a2670 COMMAND: "ksoftirqd/0"
CPU 1
CURRENT: PID: 0 TASK: ffff8802221e2670 COMMAND: "swapper/1"
RT PRIO_ARRAY: ffff880426fd4590
[no tasks queued]
CFS RB_ROOT: ffff880227018070
[no tasks queued]
CPU 2
CURRENT: PID: 0 TASK: ffff880222220000 COMMAND: "swapper/2"
RT PRIO_ARRAY: ffff8802279d4590
[no tasks queued]
CFS RB_ROOT: ffff880227018070
[no tasks queued]
CPU 3
CURRENT: PID: 11615 TASK: ffff88020c50a670 COMMAND: "runtest.sh"
RT PRIO_ARRAY: ffff8804271d4590
[no tasks queued]
CFS RB_ROOT: ffff880227018070
GROUP CFS RB_ROOT: ffff88041e0d2790
[no tasks queued]
CPU 4
CURRENT: PID: 0 TASK: ffff880222250000 COMMAND: "swapper/4"
RT PRIO_ARRAY: ffff880227bd4590
[no tasks queued]
CFS RB_ROOT: ffff880227018070
[no tasks queued]
CPU 5
CURRENT: PID: 0 TASK: ffff880222252670 COMMAND: "swapper/5"
RT PRIO_ARRAY: ffff8804273d4590
[no tasks queued]
CFS RB_ROOT: ffff880227018070
[no tasks queued]
CPU 6
CURRENT: PID: 1055 TASK: ffff88020c620000 COMMAND: "rsyslogd"
RT PRIO_ARRAY: ffff880227dd4590
[no tasks queued]
CFS RB_ROOT: ffff880227018070
GROUP CFS RB_ROOT: ffff88021df3a790 <system>
GROUP CFS RB_ROOT: ffff88020d556c78 <rsyslog.service>
[no tasks queued]
CPU 7
CURRENT: PID: 33 TASK: ffff880421d48000 COMMAND: "migration/7"
RT PRIO_ARRAY: ffff8804275d4590
[no tasks queued]
CFS RB_ROOT: ffff880227018070
[120] PID: 10653 TASK: ffff88041ead0000 COMMAND: "kworker/7:2"
*** glibc detected *** /var/CVS/crash-6.1.0/crash: free(): invalid pointer:
0x0000000004e2c210 ***
======= Backtrace: =========
/lib64/libc.so.6[0x33ff87bfee]
/var/CVS/crash-6.1.0/crash(freebuf+0x29a)[0x46f327]
/var/CVS/crash-6.1.0/crash[0x4cae70]
/var/CVS/crash-6.1.0/crash[0x4ce4b2]
/var/CVS/crash-6.1.0/crash(cmd_runq+0xfb)[0x4c9b5b]
/var/CVS/crash-6.1.0/crash(exec_command+0x20a)[0x45db7f]
/var/CVS/crash-6.1.0/crash(main_loop+0x281)[0x45d973]
/var/CVS/crash-6.1.0/crash[0x5ad159]
/var/CVS/crash-6.1.0/crash(catch_errors+0x5e)[0x5ac59e]
/var/CVS/crash-6.1.0/crash[0x5adde6]
/var/CVS/crash-6.1.0/crash(catch_errors+0x5e)[0x5ac59e]
/var/CVS/crash-6.1.0/crash(gdb_main+0x24)[0x5ae0b4]
/var/CVS/crash-6.1.0/crash(gdb_main_entry+0x2e)[0x5ae0ee]
/var/CVS/crash-6.1.0/crash(gdb_main_loop+0x263)[0x4ed248]
/var/CVS/crash-6.1.0/crash(main+0x268d)[0x45d6de]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x33ff821735]
/var/CVS/crash-6.1.0/crash[0x45af5d]
======= Memory map: ========
00400000-00a62000 r-xp 00000000 fd:01 3021564
/var/CVS/crash-6.1.0/crash
00c61000-00c85000 rw-p 00661000 fd:01 3021564
/var/CVS/crash-6.1.0/crash
00c85000-00e66000 rw-p 00000000 00:00 0
017ae000-05b7b000 rw-p 00000000 00:00 0 [heap]
33ff400000-33ff420000 r-xp 00000000 fd:01 1869596
/usr/lib64/ld-2.15.so
33ff61f000-33ff620000 r--p 0001f000 fd:01 1869596
/usr/lib64/ld-2.15.so
33ff620000-33ff621000 rw-p 00020000 fd:01 1869596
/usr/lib64/ld-2.15.so
33ff621000-33ff622000 rw-p 00000000 00:00 0
33ff800000-33ff9ac000 r-xp 00000000 fd:01 1869597
/usr/lib64/libc-2.15.so
33ff9ac000-33ffbac000 ---p 001ac000 fd:01 1869597
/usr/lib64/libc-2.15.so
33ffbac000-33ffbb0000 r--p 001ac000 fd:01 1869597
/usr/lib64/libc-2.15.so
33ffbb0000-33ffbb2000 rw-p 001b0000 fd:01 1869597
/usr/lib64/libc-2.15.so
33ffbb2000-33ffbb7000 rw-p 00000000 00:00 0
3400000000-3400003000 r-xp 00000000 fd:01 1838776
/usr/lib64/libdl-2.15.so
3400003000-3400202000 ---p 00003000 fd:01 1838776
/usr/lib64/libdl-2.15.so
3400202000-3400203000 r--p 00002000 fd:01 1838776
/usr/lib64/libdl-2.15.so
3400203000-3400204000 rw-p 00003000 fd:01 1838776
/usr/lib64/libdl-2.15.so
3400800000-3400817000 r-xp 00000000 fd:01 1869608
/usr/lib64/libz.so.1.2.5
3400817000-3400a16000 ---p 00017000 fd:01 1869608
/usr/lib64/libz.so.1.2.5
3400a16000-3400a17000 rw-p 00016000 fd:01 1869608
/usr/lib64/libz.so.1.2.5
3400c00000-3400cfa000 r-xp 00000000 fd:01 1869598
/usr/lib64/libm-2.15.so
3400cfa000-3400ef9000 ---p 000fa000 fd:01 1869598
/usr/lib64/libm-2.15.so
3400ef9000-3400efa000 r--p 000f9000 fd:01 1869598
/usr/lib64/libm-2.15.so
3400efa000-3400efb000 rw-p 000fa000 fd:01 1869598
/usr/lib64/libm-2.15.so
340f800000-340f825000 r-xp 00000000 fd:01 1869641
/usr/lib64/libtinfo.so.5.9
340f825000-340fa24000 ---p 00025000 fd:01 1869641
/usr/lib64/libtinfo.so.5.9
340fa24000-340fa28000 r--p 00024000 fd:01 1869641
/usr/lib64/libtinfo.so.5.9
340fa28000-340fa29000 rw-p 00028000 fd:01 1869641
/usr/lib64/libtinfo.so.5.9
3414800000-3414823000 r-xp 00000000 fd:01 1869642
/usr/lib64/libncurses.so.5.9
3414823000-3414a22000 ---p 00023000 fd:01 1869642
/usr/lib64/libncurses.so.5.9
3414a22000-3414a23000 r--p 00022000 fd:01 1869642
/usr/lib64/libncurses.so.5.9
3414a23000-3414a24000 rw-p 00023000 fd:01 1869642
/usr/lib64/libncurses.so.5.9
3ef8c00000-3ef8c15000 r-xp 00000000 fd:01 1842976
/usr/lib64/libgcc_s-4.7.2-20120921.so.1
3ef8c15000-3ef8e14000 ---p 00015000 fd:01 1842976
/usr/lib64/libgcc_s-4.7.2-20120921.so.1
3ef8e14000-3ef8e15000 rw-p 00014000 fd:01 1842976
/usr/lib64/libgcc_s-4.7.2-20120921.so.1
7fa098c20000-7fa099ce4000 r--p 0705e000 fd:01 3014793
/var/tmp/vmlinux.gz_pnPD6v (deleted)
7fa099ce4000-7fa09a9a5000 rw-p 00000000 00:00 0
7fa09b337000-7fa09b50c000 r--p 06e8a000 fd:01 3014793
/var/tmp/vmlinux.gz_pnPD6v (deleted)
7fa09b50c000-7fa09b9fe000 r--p 06814000 fd:01 3014793
/var/tmp/vmlinux.gz_pnPD6v (deleted)
7fa09b9fe000-7fa09bd46000 r--p 08121000 fd:01 3014793
/var/tmp/vmlinux.gz_pnPD6v (deleted)
7fa09bd46000-7fa09bf93000 r--p 065c8000 fd:01 3014793
/var/tmp/vmlinux.gz_pnPD6v (deleted)
7fa09bf93000-7fa0a100c000 r--p 01550000 fd:01 3014793
/var/tmp/vmlinux.gz_pnPD6v (deleted)
7fa0a100c000-7fa0a1298000 r--p 08468000 fd:01 3014793
/var/tmp/vmlinux.gz_pnPD6v (deleted)
7fa0a1298000-7fa0a25d4000 rw-p 00000000 00:00 0
7fa0a26f8000-7fa0a29d1000 rw-p 00000000 00:00 0
7fa0a2a63000-7fa0a8e90000 r--p 00000000 fd:01 1869461
/usr/lib/locale/locale-archive
7fa0a8e90000-7fa0a90af000 rw-p 00000000 00:00 0
7fa0a90bd000-7fa0a90be000 rw-p 00000000 00:00 0
7fa0a90be000-7fa0a90c5000 r--s 00000000 fd:01 1968224
/usr/lib64/gconv/gconv-modules.cache
7fa0a90c5000-7fa0a90cb000 rw-p 00000000 00:00 0
7fff15db3000-7fff15dd4000 rw-p 00000000 00:00 0 [stack]
7fff15dff000-7fff15e00000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted (core dumped)
$
Here's the gdb trace of the core above:
$ gdb ./crash core.19908
GNU gdb (GDB) Fedora (7.4.50.20120120-52.fc17)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /var/CVS/crash-6.1.0/crash...done.
[New LWP 19908]
Core was generated by `./crash vmcore vmlinux.gz'.
Program terminated with signal 6, Aborted.
#0 0x00000033ff835925 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.15-57.fc17.x86_64
libgcc-4.7.2-2.fc17.x86_64 ncurses-libs-5.9-4.20120204.fc17.x86_64
zlib-1.2.5-7.fc17.x86_64
(gdb) bt
#0 0x00000033ff835925 in raise () from /lib64/libc.so.6
#1 0x00000033ff8370d8 in abort () from /lib64/libc.so.6
#2 0x00000033ff874e6b in __libc_message () from /lib64/libc.so.6
#3 0x00000033ff87bfee in _int_free () from /lib64/libc.so.6
#4 0x000000000046f327 in freebuf (addr=0x3f234a0
"systemd-stdout-syslog-bridge.service") at tools.c:4710
#5 0x00000000004cae70 in free_task_group_info_array () at task.c:7578
#6 0x00000000004ce4b2 in dump_tasks_by_task_group () at task.c:8462
#7 0x00000000004c9b5b in cmd_runq () at task.c:7158
#8 0x000000000045db7f in exec_command () at main.c:764
#9 0x000000000045d973 in main_loop () at main.c:712
#10 0x00000000005ad159 in captured_command_loop (data=data@entry=0x0) at ./main.c:228
#11 0x00000000005ac59e in catch_errors (func=func@entry=0x5ad150
<captured_command_loop>,
func_args=func_args@entry=0x0, errstring=errstring@entry=0x8caaeb "",
mask=mask@entry=6) at exceptions.c:531
#12 0x00000000005adde6 in captured_main (data=data@entry=0x7fff3c694000) at ./main.c:958
#13 0x00000000005ac59e in catch_errors (func=func@entry=0x5ad230 <captured_main>,
func_args=func_args@entry=0x7fff3c694000, errstring=errstring@entry=0x8caaeb
"", mask=mask@entry=6)
at exceptions.c:531
#14 0x00000000005ae0b4 in gdb_main (args=args@entry=0x7fff3c694000) at ./main.c:973
#15 0x00000000005ae0ee in gdb_main_entry (argc=<optimized out>, argv=<optimized
out>) at ./main.c:993
#16 0x00000000004ed248 in gdb_main_loop (argc=3, argv=0x7fff3c694188) at
gdb_interface.c:76
#17 0x000000000045d6de in main (argc=3, argv=0x7fff3c694188) at main.c:617
(gdb) q
In a quick debugging session of your free_task_group_info_array()
I printed out the addresses being FREEBUF()'d, and I noted that
there were numerous instances of the same address being free twice:
static void
free_task_group_info_array(void)
{
int i;
for (i = 0; i < tgi_p; i++) {
if (tgi_array[i]->name)
FREEBUF(tgi_array[i]->name);
FREEBUF(tgi_array[i]);
}
tgi_p = 0;
FREEBUF(tgi_array);
}
I put one of the failing vmlinux/vmcore pairs here for you
to debug:
http://people.redhat.com/anderson/zhangyanfei
Secondly, another question I have is the meaning of the command's output.
First, consider this "runq" output:
crash> runq
CPU 0 RUNQUEUE: ffff8800090436c0
CURRENT: PID: 588 TASK: ffff88007e4877a0 COMMAND: "udevd"
RT PRIO_ARRAY: ffff8800090437c8
[no tasks queued]
CFS RB_ROOT: ffff880009043740
[118] PID: 2110 TASK: ffff88007d470860 COMMAND: "check-cdrom.sh"
[118] PID: 2109 TASK: ffff88007f1247a0 COMMAND: "check-cdrom.sh"
[118] PID: 2114 TASK: ffff88007f20e080 COMMAND: "udevd"
CPU 1 RUNQUEUE: ffff88000905b6c0
CURRENT: PID: 2113 TASK: ffff88007e8ac140 COMMAND: "udevd"
RT PRIO_ARRAY: ffff88000905b7c8
[no tasks queued]
CFS RB_ROOT: ffff88000905b740
[118] PID: 2092 TASK: ffff88007d7a4760 COMMAND: "MAKEDEV"
[118] PID: 1983 TASK: ffff88007e59f140 COMMAND: "udevd"
[118] PID: 2064 TASK: ffff88007e40f7a0 COMMAND: "udevd"
[115] PID: 2111 TASK: ffff88007e4278a0 COMMAND: "kthreadd"
crash>
In the above case, the per-cpu "rq" structure addresses are shown as:
CPU 0 RUNQUEUE: ffff8800090436c0
CPU 1 RUNQUEUE: ffff88000905b6c0
And embedded in each of the rq structures above are these two rb_root
structures:
CFS RB_ROOT: ffff880009043740 (embedded in rq @ffff8800090436c0)
CFS RB_ROOT: ffff88000905b740 (embedded in rq @ffff88000905b6c0)
And starting at those rb_root structures, the tree of tasks are dumped.
Now, your "runq -q" option doesn't show any "starting point"
structure
address, but rather they just show "CPU 0" and "CPU 1":
crash> runq -g
CPU 0
CURRENT: PID: 588 TASK: ffff88007e4877a0 COMMAND: "udevd"
RT PRIO_ARRAY: ffff8800090437c8
[no tasks queued]
CFS RB_ROOT: ffff880009093548
[118] PID: 2110 TASK: ffff88007d470860 COMMAND: "check-cdrom.sh"
[118] PID: 2109 TASK: ffff88007f1247a0 COMMAND: "check-cdrom.sh"
[118] PID: 2114 TASK: ffff88007f20e080 COMMAND: "udevd"
CPU 1
CURRENT: PID: 2113 TASK: ffff88007e8ac140 COMMAND: "udevd"
RT PRIO_ARRAY: ffff88000905b7c8
[no tasks queued]
CFS RB_ROOT: ffff880009093548
[118] PID: 2092 TASK: ffff88007d7a4760 COMMAND: "MAKEDEV"
[118] PID: 1983 TASK: ffff88007e59f140 COMMAND: "udevd"
[118] PID: 2064 TASK: ffff88007e40f7a0 COMMAND: "udevd"
[115] PID: 2111 TASK: ffff88007e4278a0 COMMAND: "kthreadd"
crash>
I would think that there might be a useful address of a per-cpu
structure that could be shown there as well?
And secondly, I'm confused as to why the "CFS RB_ROOT" address for
all cpus is the same address -- for example, above they are both at
ffff880009093548. How can the two rb trees have the same rb_root?
Dave
9:37 p.m.
于 2012年11月08日 03:15, Dave Anderson 写道:
----- Original Message -----
>
> ok. I rewrite the patch and they are tested ok in my box.
>
> Thanks
> Zhang
My tests weren't so successful this time, and I also have some questions
about the runq -g output.
I tested your latest patches on a sample set of 70 dumpfiles whose
kernels all use CFS runqueues. In 7 of the 70 "runq -g" tests,
the command caused the crash session to fail like so:
<snip>
In a quick debugging session of your free_task_group_info_array()
I printed out the addresses being FREEBUF()'d, and I noted that
there were numerous instances of the same address being free twice:
static void
free_task_group_info_array(void)
{
int i;
for (i = 0; i < tgi_p; i++) {
if (tgi_array[i]->name)
FREEBUF(tgi_array[i]->name);
FREEBUF(tgi_array[i]);
}
tgi_p = 0;
FREEBUF(tgi_array);
}
I put one of the failing vmlinux/vmcore pairs here for you
to debug:
http://people.redhat.com/anderson/zhangyanfei
This is so weird. In my test on the vmcore you provided, 'runq -g' ran well
for the first time and caused the crash session to fail the next time.
From the debug information above and from my tests, I noticed that it
always
failed on the same place when FREEBUF a name. So I checked the function
get_task_group_name and changed the way to return a name buf. Now the command
works well on the vmcore.
Secondly, another question I have is the meaning of the command's output.
First, consider this "runq" output:
crash> runq
CPU 0 RUNQUEUE: ffff8800090436c0
CURRENT: PID: 588 TASK: ffff88007e4877a0 COMMAND: "udevd"
RT PRIO_ARRAY: ffff8800090437c8
[no tasks queued]
CFS RB_ROOT: ffff880009043740
[118] PID: 2110 TASK: ffff88007d470860 COMMAND: "check-cdrom.sh"
[118] PID: 2109 TASK: ffff88007f1247a0 COMMAND: "check-cdrom.sh"
[118] PID: 2114 TASK: ffff88007f20e080 COMMAND: "udevd"
CPU 1 RUNQUEUE: ffff88000905b6c0
CURRENT: PID: 2113 TASK: ffff88007e8ac140 COMMAND: "udevd"
RT PRIO_ARRAY: ffff88000905b7c8
[no tasks queued]
CFS RB_ROOT: ffff88000905b740
[118] PID: 2092 TASK: ffff88007d7a4760 COMMAND: "MAKEDEV"
[118] PID: 1983 TASK: ffff88007e59f140 COMMAND: "udevd"
[118] PID: 2064 TASK: ffff88007e40f7a0 COMMAND: "udevd"
[115] PID: 2111 TASK: ffff88007e4278a0 COMMAND: "kthreadd"
crash>
In the above case, the per-cpu "rq" structure addresses are shown as:
CPU 0 RUNQUEUE: ffff8800090436c0
CPU 1 RUNQUEUE: ffff88000905b6c0
And embedded in each of the rq structures above are these two rb_root
structures:
CFS RB_ROOT: ffff880009043740 (embedded in rq @ffff8800090436c0)
CFS RB_ROOT: ffff88000905b740 (embedded in rq @ffff88000905b6c0)
And starting at those rb_root structures, the tree of tasks are dumped.
Now, your "runq -q" option doesn't show any "starting point"
structure
address, but rather they just show "CPU 0" and "CPU 1":
crash> runq -g
CPU 0
CURRENT: PID: 588 TASK: ffff88007e4877a0 COMMAND: "udevd"
RT PRIO_ARRAY: ffff8800090437c8
[no tasks queued]
CFS RB_ROOT: ffff880009093548
[118] PID: 2110 TASK: ffff88007d470860 COMMAND: "check-cdrom.sh"
[118] PID: 2109 TASK: ffff88007f1247a0 COMMAND: "check-cdrom.sh"
[118] PID: 2114 TASK: ffff88007f20e080 COMMAND: "udevd"
CPU 1
CURRENT: PID: 2113 TASK: ffff88007e8ac140 COMMAND: "udevd"
RT PRIO_ARRAY: ffff88000905b7c8
[no tasks queued]
CFS RB_ROOT: ffff880009093548
[118] PID: 2092 TASK: ffff88007d7a4760 COMMAND: "MAKEDEV"
[118] PID: 1983 TASK: ffff88007e59f140 COMMAND: "udevd"
[118] PID: 2064 TASK: ffff88007e40f7a0 COMMAND: "udevd"
[115] PID: 2111 TASK: ffff88007e4278a0 COMMAND: "kthreadd"
crash>
I would think that there might be a useful address of a per-cpu
structure that could be shown there as well?
OK, this is added.
And secondly, I'm confused as to why the "CFS RB_ROOT" address for
all cpus is the same address -- for example, above they are both at
ffff880009093548. How can the two rb trees have the same rb_root?
My neglect, sorry. fixed.
Thanks
Zhang
2:15 p.m.
----- Original Message -----
> Secondly, another question I have is the meaning of the
command's output.
>
> First, consider this "runq" output:
>
> crash> runq
> CPU 0 RUNQUEUE: ffff8800090436c0
> CURRENT: PID: 588 TASK: ffff88007e4877a0 COMMAND: "udevd"
> RT PRIO_ARRAY: ffff8800090437c8
> [no tasks queued]
> CFS RB_ROOT: ffff880009043740
> [118] PID: 2110 TASK: ffff88007d470860 COMMAND: "check-cdrom.sh"
> [118] PID: 2109 TASK: ffff88007f1247a0 COMMAND: "check-cdrom.sh"
> [118] PID: 2114 TASK: ffff88007f20e080 COMMAND: "udevd"
>
> CPU 1 RUNQUEUE: ffff88000905b6c0
> CURRENT: PID: 2113 TASK: ffff88007e8ac140 COMMAND: "udevd"
> RT PRIO_ARRAY: ffff88000905b7c8
> [no tasks queued]
> CFS RB_ROOT: ffff88000905b740
> [118] PID: 2092 TASK: ffff88007d7a4760 COMMAND: "MAKEDEV"
> [118] PID: 1983 TASK: ffff88007e59f140 COMMAND: "udevd"
> [118] PID: 2064 TASK: ffff88007e40f7a0 COMMAND: "udevd"
> [115] PID: 2111 TASK: ffff88007e4278a0 COMMAND: "kthreadd"
> crash>
>
> In the above case, the per-cpu "rq" structure addresses are shown as:
>
> CPU 0 RUNQUEUE: ffff8800090436c0
> CPU 1 RUNQUEUE: ffff88000905b6c0
>
> And embedded in each of the rq structures above are these two rb_root
> structures:
>
> CFS RB_ROOT: ffff880009043740 (embedded in rq @ffff8800090436c0)
> CFS RB_ROOT: ffff88000905b740 (embedded in rq @ffff88000905b6c0)
>
> And starting at those rb_root structures, the tree of tasks are
> dumped.
>
> Now, your "runq -q" option doesn't show any "starting point"
structure
> address, but rather they just show "CPU 0" and "CPU 1":
>
> crash> runq -g
> CPU 0
> CURRENT: PID: 588 TASK: ffff88007e4877a0 COMMAND: "udevd"
> RT PRIO_ARRAY: ffff8800090437c8
> [no tasks queued]
> CFS RB_ROOT: ffff880009093548
> [118] PID: 2110 TASK: ffff88007d470860 COMMAND: "check-cdrom.sh"
> [118] PID: 2109 TASK: ffff88007f1247a0 COMMAND: "check-cdrom.sh"
> [118] PID: 2114 TASK: ffff88007f20e080 COMMAND: "udevd"
>
> CPU 1
> CURRENT: PID: 2113 TASK: ffff88007e8ac140 COMMAND: "udevd"
> RT PRIO_ARRAY: ffff88000905b7c8
> [no tasks queued]
> CFS RB_ROOT: ffff880009093548
> [118] PID: 2092 TASK: ffff88007d7a4760 COMMAND: "MAKEDEV"
> [118] PID: 1983 TASK: ffff88007e59f140 COMMAND: "udevd"
> [118] PID: 2064 TASK: ffff88007e40f7a0 COMMAND: "udevd"
> [115] PID: 2111 TASK: ffff88007e4278a0 COMMAND: "kthreadd"
> crash>
>
> I would think that there might be a useful address of a per-cpu
> structure that could be shown there as well?
OK, this is added.
OK, but now your addition just mimics the "runq" output. You are just
calculating the CFS "runq" address for display, but that address does
not passed to (or used by) either dump_tasks_in_task_group_rt_rq()
or dump_tasks_in_task_group_cfs_rq().
I was suggesting that you display per-cpu addresses generated from the
init_task_group/root_task_group task_group structures. Wouldn't it make
more sense for it to show both the per-cpu "rt_rq" and "cfs_rq"
addresses?
In other words, maybe something like:
CPU 0 TASK_GROUP RT_RQ: ffff88021e213e28 TASK_GROUP CFS_RQ: ffff88021e213d30
CURRENT: ...
Aside from that, the latest patch passes my tests OK.
Thanks,
Dave
2:23 a.m.
于 2012年11月13日 04:15, Dave Anderson 写道:
----- Original Message -----
>> Secondly, another question I have is the meaning of the command's output.
>>
>> First, consider this "runq" output:
>>
>> crash> runq
>> CPU 0 RUNQUEUE: ffff8800090436c0
>> CURRENT: PID: 588 TASK: ffff88007e4877a0 COMMAND: "udevd"
>> RT PRIO_ARRAY: ffff8800090437c8
>> [no tasks queued]
>> CFS RB_ROOT: ffff880009043740
>> [118] PID: 2110 TASK: ffff88007d470860 COMMAND:
"check-cdrom.sh"
>> [118] PID: 2109 TASK: ffff88007f1247a0 COMMAND:
"check-cdrom.sh"
>> [118] PID: 2114 TASK: ffff88007f20e080 COMMAND: "udevd"
>>
>> CPU 1 RUNQUEUE: ffff88000905b6c0
>> CURRENT: PID: 2113 TASK: ffff88007e8ac140 COMMAND: "udevd"
>> RT PRIO_ARRAY: ffff88000905b7c8
>> [no tasks queued]
>> CFS RB_ROOT: ffff88000905b740
>> [118] PID: 2092 TASK: ffff88007d7a4760 COMMAND: "MAKEDEV"
>> [118] PID: 1983 TASK: ffff88007e59f140 COMMAND: "udevd"
>> [118] PID: 2064 TASK: ffff88007e40f7a0 COMMAND: "udevd"
>> [115] PID: 2111 TASK: ffff88007e4278a0 COMMAND: "kthreadd"
>> crash>
>>
>> In the above case, the per-cpu "rq" structure addresses are shown as:
>>
>> CPU 0 RUNQUEUE: ffff8800090436c0
>> CPU 1 RUNQUEUE: ffff88000905b6c0
>>
>> And embedded in each of the rq structures above are these two rb_root
>> structures:
>>
>> CFS RB_ROOT: ffff880009043740 (embedded in rq @ffff8800090436c0)
>> CFS RB_ROOT: ffff88000905b740 (embedded in rq @ffff88000905b6c0)
>>
>> And starting at those rb_root structures, the tree of tasks are
>> dumped.
>>
>> Now, your "runq -q" option doesn't show any "starting
point" structure
>> address, but rather they just show "CPU 0" and "CPU 1":
>>
>> crash> runq -g
>> CPU 0
>> CURRENT: PID: 588 TASK: ffff88007e4877a0 COMMAND: "udevd"
>> RT PRIO_ARRAY: ffff8800090437c8
>> [no tasks queued]
>> CFS RB_ROOT: ffff880009093548
>> [118] PID: 2110 TASK: ffff88007d470860 COMMAND:
"check-cdrom.sh"
>> [118] PID: 2109 TASK: ffff88007f1247a0 COMMAND:
"check-cdrom.sh"
>> [118] PID: 2114 TASK: ffff88007f20e080 COMMAND: "udevd"
>>
>> CPU 1
>> CURRENT: PID: 2113 TASK: ffff88007e8ac140 COMMAND: "udevd"
>> RT PRIO_ARRAY: ffff88000905b7c8
>> [no tasks queued]
>> CFS RB_ROOT: ffff880009093548
>> [118] PID: 2092 TASK: ffff88007d7a4760 COMMAND: "MAKEDEV"
>> [118] PID: 1983 TASK: ffff88007e59f140 COMMAND: "udevd"
>> [118] PID: 2064 TASK: ffff88007e40f7a0 COMMAND: "udevd"
>> [115] PID: 2111 TASK: ffff88007e4278a0 COMMAND: "kthreadd"
>> crash>
>>
>> I would think that there might be a useful address of a per-cpu
>> structure that could be shown there as well?
>
> OK, this is added.
OK, but now your addition just mimics the "runq" output. You are just
calculating the CFS "runq" address for display, but that address does
not passed to (or used by) either dump_tasks_in_task_group_rt_rq()
or dump_tasks_in_task_group_cfs_rq().
I was suggesting that you display per-cpu addresses generated from the
init_task_group/root_task_group task_group structures. Wouldn't it make
more sense for it to show both the per-cpu "rt_rq" and "cfs_rq"
addresses?
In other words, maybe something like:
CPU 0 TASK_GROUP RT_RQ: ffff88021e213e28 TASK_GROUP CFS_RQ: ffff88021e213d30
CURRENT: ...
OK, fixed.
Thanks
Zhang
12:56 p.m.
----- Original Message -----
> I was suggesting that you display per-cpu addresses generated
from the
> init_task_group/root_task_group task_group structures. Wouldn't it make
> more sense for it to show both the per-cpu "rt_rq" and "cfs_rq"
> addresses?
>
> In other words, maybe something like:
>
> CPU 0 TASK_GROUP RT_RQ: ffff88021e213e28 TASK_GROUP CFS_RQ:
> ffff88021e213d30
> CURRENT: ...
>
OK, fixed.
Hello Zhang,
Thanks -- and this patch tests OK.
For clarity, I changed the output slightly to show the per-cpu task_group rt_rq
and cfs_rq pointers above their respective queues. (The rt_rq and rt prio_array
addresses are the same because of the structure layout, but that's not
necessarily always going to be true.) And I tried to make the help page
description a little cleaner. So here's what it looks like:
crash> help runq
... [ cut ] ...
-g Display tasks hierarchically by task_group. Tasks in throttled groups
are also displayed.
... [cut] ...
Display tasks hierarchically by task_group:
crash> runq -g
CPU 0
CURRENT: PID: 14734 TASK: ffff88010626f500 COMMAND: "sh"
TASK_GROUP RT_RQ: ffff880028216808
RT PRIO_ARRAY: ffff880028216808
[ 0] GROUP RT PRIO_ARRAY: ffff880139fc9800 <test1> (THROTTLED)
[ 0] PID: 14750 TASK: ffff88013a4dd540 COMMAND: "rtloop99"
[ 1] PID: 14748 TASK: ffff88013bbca040 COMMAND: "rtloop98"
[ 1] GROUP RT PRIO_ARRAY: ffff880089029000 <test11>
[ 1] PID: 14752 TASK: ffff880088abf500 COMMAND: "rtloop98"
[ 54] PID: 14749 TASK: ffff880037a4e080 COMMAND: "rtloop45"
[ 98] PID: 14746 TASK: ffff88012678c080 COMMAND: "rtloop1"
TASK_GROUP CFS_RQ: ffff8800282166e8
CFS RB_ROOT: ffff880028216718
[120] PID: 14740 TASK: ffff88013b1e6080 COMMAND: "sh"
[120] PID: 14738 TASK: ffff88012678d540 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff8800897af430 <test2> (THROTTLED)
[120] PID: 14732 TASK: ffff88013bbcb500 COMMAND: "sh"
[120] PID: 14728 TASK: ffff8800b3496080 COMMAND: "sh"
[120] PID: 14730 TASK: ffff880037833540 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff880037943e30 <test1> (THROTTLED)
[120] PID: 14726 TASK: ffff880138d42aa0 COMMAND: "sh"
CPU 1
CURRENT: PID: 3269 TASK: ffff88013b0fa040 COMMAND: "bash"
TASK_GROUP RT_RQ: ffff880028296808
RT PRIO_ARRAY: ffff880028296808
[ 0] GROUP RT PRIO_ARRAY: ffff88008a1f5000 <test1> (THROTTLED)
[ 0] GROUP RT PRIO_ARRAY: ffff880121774800 <test11>
[ 0] PID: 14753 TASK: ffff88013bbbaae0 COMMAND:
"rtloop99"
[ 98] PID: 14745 TASK: ffff880126763500 COMMAND: "rtloop1"
[ 98] PID: 14747 TASK: ffff88013b1e6ae0 COMMAND: "rtloop1"
TASK_GROUP CFS_RQ: ffff8800282966e8
CFS RB_ROOT: ffff880028296718
GROUP CFS RB_ROOT: ffff8800896eac30 <test1>
[120] PID: 14724 TASK: ffff880139632080 COMMAND: "sh"
[120] PID: 14742 TASK: ffff880126762aa0 COMMAND: "sh"
[120] PID: 14736 TASK: ffff88010626e040 COMMAND: "sh"
crash>
Are you planning any further changes, or would you like to go with this
version for crash-6.1.1?
Dave
1:27 a.m.
于 2012年11月14日 02:56, Dave Anderson 写道:
----- Original Message -----
>> I was suggesting that you display per-cpu addresses generated from the
>> init_task_group/root_task_group task_group structures. Wouldn't it make
>> more sense for it to show both the per-cpu "rt_rq" and
"cfs_rq"
>> addresses?
>>
>> In other words, maybe something like:
>>
>> CPU 0 TASK_GROUP RT_RQ: ffff88021e213e28 TASK_GROUP CFS_RQ:
>> ffff88021e213d30
>> CURRENT: ...
>>
>
> OK, fixed.
Hello Zhang,
Thanks -- and this patch tests OK.
For clarity, I changed the output slightly to show the per-cpu task_group rt_rq
and cfs_rq pointers above their respective queues. (The rt_rq and rt prio_array
addresses are the same because of the structure layout, but that's not
necessarily always going to be true.) And I tried to make the help page
description a little cleaner. So here's what it looks like:
crash> help runq
... [ cut ] ...
-g Display tasks hierarchically by task_group. Tasks in throttled groups
are also displayed.
... [cut] ...
Display tasks hierarchically by task_group:
crash> runq -g
CPU 0
CURRENT: PID: 14734 TASK: ffff88010626f500 COMMAND: "sh"
TASK_GROUP RT_RQ: ffff880028216808
RT PRIO_ARRAY: ffff880028216808
[ 0] GROUP RT PRIO_ARRAY: ffff880139fc9800 <test1> (THROTTLED)
[ 0] PID: 14750 TASK: ffff88013a4dd540 COMMAND: "rtloop99"
[ 1] PID: 14748 TASK: ffff88013bbca040 COMMAND: "rtloop98"
[ 1] GROUP RT PRIO_ARRAY: ffff880089029000 <test11>
[ 1] PID: 14752 TASK: ffff880088abf500 COMMAND:
"rtloop98"
[ 54] PID: 14749 TASK: ffff880037a4e080 COMMAND: "rtloop45"
[ 98] PID: 14746 TASK: ffff88012678c080 COMMAND: "rtloop1"
TASK_GROUP CFS_RQ: ffff8800282166e8
CFS RB_ROOT: ffff880028216718
[120] PID: 14740 TASK: ffff88013b1e6080 COMMAND: "sh"
[120] PID: 14738 TASK: ffff88012678d540 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff8800897af430 <test2> (THROTTLED)
[120] PID: 14732 TASK: ffff88013bbcb500 COMMAND: "sh"
[120] PID: 14728 TASK: ffff8800b3496080 COMMAND: "sh"
[120] PID: 14730 TASK: ffff880037833540 COMMAND: "sh"
GROUP CFS RB_ROOT: ffff880037943e30 <test1> (THROTTLED)
[120] PID: 14726 TASK: ffff880138d42aa0 COMMAND: "sh"
CPU 1
CURRENT: PID: 3269 TASK: ffff88013b0fa040 COMMAND: "bash"
TASK_GROUP RT_RQ: ffff880028296808
RT PRIO_ARRAY: ffff880028296808
[ 0] GROUP RT PRIO_ARRAY: ffff88008a1f5000 <test1> (THROTTLED)
[ 0] GROUP RT PRIO_ARRAY: ffff880121774800 <test11>
[ 0] PID: 14753 TASK: ffff88013bbbaae0 COMMAND:
"rtloop99"
[ 98] PID: 14745 TASK: ffff880126763500 COMMAND: "rtloop1"
[ 98] PID: 14747 TASK: ffff88013b1e6ae0 COMMAND: "rtloop1"
TASK_GROUP CFS_RQ: ffff8800282966e8
CFS RB_ROOT: ffff880028296718
GROUP CFS RB_ROOT: ffff8800896eac30 <test1>
[120] PID: 14724 TASK: ffff880139632080 COMMAND: "sh"
[120] PID: 14742 TASK: ffff880126762aa0 COMMAND: "sh"
[120] PID: 14736 TASK: ffff88010626e040 COMMAND: "sh"
crash>
This seems more clear. Thanks.
Are you planning any further changes, or would you like to go with this
version for crash-6.1.1?
No more changes for runq -g currently. So I am happy with the patch accepted.
And once I said in the mail that I will change the rt_rq display by runq
with no argument non-hierarchically and remove the group info in it. So
I made the patch. And the patch is based on the two patches for runq -g.
Thanks
Zhang
1:45 p.m.
----- Original Message -----
> For clarity, I changed the output slightly to show the per-cpu
task_group rt_rq
> and cfs_rq pointers above their respective queues. (The rt_rq and rt prio_array
> addresses are the same because of the structure layout, but that's not
> necessarily always going to be true.) And I tried to make the help page
> description a little cleaner. So here's what it looks like:
>
> crash> help runq
> ... [ cut ] ...
>
> -g Display tasks hierarchically by task_group. Tasks in
> throttled groups
> are also displayed.
>
> ... [cut] ...
>
> Display tasks hierarchically by task_group:
>
> crash> runq -g
> CPU 0
> CURRENT: PID: 14734 TASK: ffff88010626f500 COMMAND: "sh"
> TASK_GROUP RT_RQ: ffff880028216808
> RT PRIO_ARRAY: ffff880028216808
> [ 0] GROUP RT PRIO_ARRAY: ffff880139fc9800 <test1> (THROTTLED)
> [ 0] PID: 14750 TASK: ffff88013a4dd540 COMMAND: "rtloop99"
> [ 1] PID: 14748 TASK: ffff88013bbca040 COMMAND: "rtloop98"
> [ 1] GROUP RT PRIO_ARRAY: ffff880089029000 <test11>
> [ 1] PID: 14752 TASK: ffff880088abf500 COMMAND:
"rtloop98"
> [ 54] PID: 14749 TASK: ffff880037a4e080 COMMAND: "rtloop45"
> [ 98] PID: 14746 TASK: ffff88012678c080 COMMAND: "rtloop1"
> TASK_GROUP CFS_RQ: ffff8800282166e8
> CFS RB_ROOT: ffff880028216718
> [120] PID: 14740 TASK: ffff88013b1e6080 COMMAND: "sh"
> [120] PID: 14738 TASK: ffff88012678d540 COMMAND: "sh"
> GROUP CFS RB_ROOT: ffff8800897af430 <test2> (THROTTLED)
> [120] PID: 14732 TASK: ffff88013bbcb500 COMMAND: "sh"
> [120] PID: 14728 TASK: ffff8800b3496080 COMMAND: "sh"
> [120] PID: 14730 TASK: ffff880037833540 COMMAND: "sh"
> GROUP CFS RB_ROOT: ffff880037943e30 <test1> (THROTTLED)
> [120] PID: 14726 TASK: ffff880138d42aa0 COMMAND: "sh"
>
> CPU 1
> CURRENT: PID: 3269 TASK: ffff88013b0fa040 COMMAND: "bash"
> TASK_GROUP RT_RQ: ffff880028296808
> RT PRIO_ARRAY: ffff880028296808
> [ 0] GROUP RT PRIO_ARRAY: ffff88008a1f5000 <test1> (THROTTLED)
> [ 0] GROUP RT PRIO_ARRAY: ffff880121774800 <test11>
> [ 0] PID: 14753 TASK: ffff88013bbbaae0 COMMAND:
"rtloop99"
> [ 98] PID: 14745 TASK: ffff880126763500 COMMAND: "rtloop1"
> [ 98] PID: 14747 TASK: ffff88013b1e6ae0 COMMAND: "rtloop1"
> TASK_GROUP CFS_RQ: ffff8800282966e8
> CFS RB_ROOT: ffff880028296718
> GROUP CFS RB_ROOT: ffff8800896eac30 <test1>
> [120] PID: 14724 TASK: ffff880139632080 COMMAND: "sh"
> [120] PID: 14742 TASK: ffff880126762aa0 COMMAND: "sh"
> [120] PID: 14736 TASK: ffff88010626e040 COMMAND: "sh"
>
> crash>
This seems more clear. Thanks.
>
> Are you planning any further changes, or would you like to go with this
> version for crash-6.1.1?
No more changes for runq -g currently. So I am happy with the patch accepted.
And once I said in the mail that I will change the rt_rq display by runq
with no argument non-hierarchically and remove the group info in it. So
I made the patch. And the patch is based on the two patches for runq -g.
This third patch also tests OK -- all 3 patches are queued for crash-6.1.1.
Thanks,
Dave
4245
days inactive
4267
days old
devel@lists.crash-utility.osci.io
21 comments
3 participants
participants (3)
-
Dave Anderson -
Zhang Yanfei -
zhangyanfei