Re: [Crash-utility] [PATCH] runq: make tasks in throttled cfs_rqs/rt_rqs displayed

Thursday, 1 November 2012

----- Original Message -----

...
 > > My suggestion re: a new -g flag would have it applicable to
both CFS and RT
 > > run queues (even though the RT groups are currently being shown).  I was
 > > thinking that it might be safest to have "runq" (with no argument) to
just
 > > display a simple list of tasks, and only show group information with -g.
 > > Are you suggesting that -g would only apply to throttled cfs_rqs/rt_rqs?
 > > 
 > 
 > nop, '-g' is ok to show all rqs with group information.
 > 
 > I rewrote the patch by adding option '-g' to runq and not changing any
behaviour of
 > the runq (with no argument).
 > 
 > TODO:
 > 1. The help info about the -g option.
 > 
 > I still have a suggestion: when we dump cfs_rq, we don't dump it
hierarchically,
 > but for the current rt_rq, we dump it hierarchically with group info.
 > So could I make another patch to remove the rt_rq hierarchy, just make it
 > displayed identical with cfs_rq display.(For -g will display group info for both
rt_rq
 > and cfs_rq)

 Yes, that's exactly what I was suggesting.

 I worry about the large number of kernel structure.member declarations that the
 command depends upon, because if just one of them changes, it breaks the command.
 So it would be preferable if the "runq" command (with no arguments) would only
use
 a minimal set of structure.member offsets.

 Thanks,
   Dave 
And to follow up, I'm still running tests (and will do so overnight) on your latest
patch, but I immediately seen this on any 2.6.30, 2.6.31 or 2.6.32 kernel, and on
some 2.6.36 and 2.6.38 kernels, where "runq -g" fails like this:

  crash> runq -g
  runq: invalid kernel virtual address: 0  type: "dentry"
  crash>

And we certainly want to keep the group information separate from
the normal "runq" command.  Here on a "live" 4 cpu Fedora 3.6.3
kernel,
the command output exceeds 1000 lines!  I'm pretty sure that most
people will *not* want to see all of this:

  crash> runq -g
  CPU 0
    CURRENT: PID: 0      TASK: ffffffff81c13420  COMMAND: "swapper/0"
    RT PRIO_ARRAY: ffff88021e213e28
       [100] GROUP RT PRIO_ARRAY: ffff88020db22000 <system>
             [100] GROUP RT PRIO_ARRAY: ffff8801f8180000 <udisks2.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8e11800 <colord-sane.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8802116f0000 <colord.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f1fd000 <cups.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f51800 <upower.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f994c800 <accounts-daemon.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880212ae1000 <prefdm.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8180800 <atd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9abe000 <crond.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9aba000 <autofs.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f54000 <sm-client.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f7ee6000 <sendmail.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f7e21800 <nscd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f7d00800 <wpa_supplicant.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f6a000 <ypbind.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f6d800 <nfs-lock.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8e12800 <rpcbind.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8d7a800 <libvirtd.service>
                   [100] GROUP RT PRIO_ARRAY: ffff8801f9944000 <libvirt>
                         [100] GROUP RT PRIO_ARRAY: ffff8801f7e22800 <lxc>
                               [no tasks queued]
                         [100] GROUP RT PRIO_ARRAY: ffff88021091b000 <qemu>
                               [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211f4e000 <sshd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211f4a800 <ksmtuned.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9a7c000 <dbus.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9a78800 <rsyslog.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f99cd800 <avahi-daemon.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f99c9800
<system-setup-keyboard.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9975800 <systemd-logind.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801fc2cb800 <irqbalance.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f98c6800 <abrt-xorg.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9885800 <NetworkManager.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020ee39800 <acpid.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020ee3d800 <gpm.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f40d800 <mcelog.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211ed9800 <abrt-oops.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f403000 <abrtd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f400800 <smartd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020dc47800 <auditd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211a55000 <iscsid.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8802108ab800 <bluetooth.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211f4a000 <fsck@.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211edc000 <udev.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880212adc000 <systemd-journald.service>
                   [no tasks queued]
    CFS RB_ROOT: ffff880215806068
       GROUP CFS RB_ROOT: ffff8801ed860728
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ed861a28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020dab6228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020dd98e28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ef1ec828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8800a07d6d28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ef2ad928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f11fe228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f180b628
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020ee24528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e110828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880210a5fe28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e161c28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020f503b28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f5b128
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ff7c28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020d972828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880210a5f328
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e115c28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f5d128
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8802129c2b28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8d6a028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f7d3e328
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f822ed28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8fbcd28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8fbde28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f822ec28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8d77d28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020dd59328
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8e7d528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8fbdb28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f59428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8e7e128
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020ee78b28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020d983f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ed4028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9b92428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9b03128
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020edd4928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9a71628
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9a70028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020dcfe128
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ec9b28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020edd5328
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020d924028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211e69228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020dd59428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211f26f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211e68528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020f75e828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020f75e528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ef0e28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020d972f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e2aac28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e110428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020db0cc28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ee1428 <system>
          GROUP CFS RB_ROOT: ffff88020ed3f928 <udisks2.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f814c028 <colord-sane.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dd0eb28 <colord.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88021041b328 <cups.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dd98228 <upower.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880210aab628 <accounts-daemon.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f7f6d728 <prefdm.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f7f6c028 <atd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d01e28 <crond.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d01728 <autofs.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d980228 <sm-client.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8f5dd28 <sendmail.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d77c28 <nscd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020e152f28 <wpa_supplicant.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d6bb28 <ypbind.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d943b28 <nfs-lock.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211ec8428 <rpcbind.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d981728 <libvirtd.service>
             GROUP CFS RB_ROOT: ffff8801f82b6828 <libvirt>
                GROUP CFS RB_ROOT: ffff8801f828d828 <lxc>
                   [no tasks queued]
                GROUP CFS RB_ROOT: ffff8801f82b7628 <qemu>
                   [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801fc232d28 <sshd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f9a70428 <ksmtuned.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020ddf7128 <dbus.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f9a77728 <rsyslog.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dd53e28 <avahi-daemon.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dcf6728 <system-setup-keyboard.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211e68c28 <systemd-logind.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211e3f928 <irqbalance.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020f638828 <abrt-xorg.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880210800c28 <NetworkManager.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dcfe928 <acpid.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d982628 <gpm.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020ff68d28 <mcelog.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d982428 <abrt-oops.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d983928 <abrtd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211a5b728 <smartd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880210af7528 <auditd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8802116f9c28 <iscsid.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020e2aa328 <bluetooth.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d942828 <fsck@.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211a5a128 <udev.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8802116f8328 <systemd-journald.service>
             [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020a844628
          [no tasks queued]

  CPU 1
    CURRENT: PID: 1122   TASK: ffff8801f8e18000  COMMAND: "Xorg"
    RT PRIO_ARRAY: ffff88021e253e28
       [100] GROUP RT PRIO_ARRAY: ffff88020db21800 <system>
             [100] GROUP RT PRIO_ARRAY: ffff8801f8187800 <udisks2.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8e10800 <colord-sane.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8802116f5800 <colord.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f1fc800 <cups.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f50000 <upower.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f994c000 <accounts-daemon.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880212ae3000 <prefdm.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8181000 <atd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9abe800 <crond.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9aba800 <autofs.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f52000 <sm-client.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f7ee5800 <sendmail.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f7e20800 <nscd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f7d01000 <wpa_supplicant.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f69800 <ypbind.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f6e000 <nfs-lock.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8e13000 <rpcbind.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8d7b000 <libvirtd.service>
                   [100] GROUP RT PRIO_ARRAY: ffff8801f9943000 <libvirt>
                         [100] GROUP RT PRIO_ARRAY: ffff8801f7e23000 <lxc>
                               [no tasks queued]
                         [100] GROUP RT PRIO_ARRAY: ffff88021091a800 <qemu>
                               [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880212bdf800 <sshd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211f4b000 <ksmtuned.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9a7c800 <dbus.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9a78000 <rsyslog.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f99ce000 <avahi-daemon.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f99ca000
<system-setup-keyboard.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9976000 <systemd-logind.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801fc2cd000 <irqbalance.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f98c7000 <abrt-xorg.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9886000 <NetworkManager.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020ee39000 <acpid.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020ee3d000 <gpm.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f40d000 <mcelog.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211ed9000 <abrt-oops.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f401000 <abrtd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f404800 <smartd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801fc2c8000 <auditd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211a53000 <iscsid.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8802108ac800 <bluetooth.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211f4d800 <fsck@.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211edc800 <udev.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880212adc800 <systemd-journald.service>
                   [no tasks queued]
    CFS RB_ROOT: ffff880215806068
       GROUP CFS RB_ROOT: ffff880211ee1b28 <system>
          GROUP CFS RB_ROOT: ffff8801f7f6d528 <prefdm.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020ed3fa28 <udisks2.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f814dc28 <colord-sane.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dd0ff28 <colord.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88021041a928 <cups.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dd98128 <upower.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880210aab928 <accounts-daemon.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f7f6d228 <atd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d01228 <crond.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d00c28 <autofs.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d980528 <sm-client.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8f5c228 <sendmail.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d76b28 <nscd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020e153028 <wpa_supplicant.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d6bc28 <ypbind.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d942d28 <nfs-lock.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801fc177928 <rpcbind.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d981d28 <libvirtd.service>
             GROUP CFS RB_ROOT: ffff8801f82b6128 <libvirt>
                GROUP CFS RB_ROOT: ffff8801f828cb28 <lxc>
                   [no tasks queued]
                GROUP CFS RB_ROOT: ffff8801f82b7228 <qemu>
                   [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801fc232f28 <sshd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f9a70928 <ksmtuned.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020ddf7828 <dbus.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f9a77828 <rsyslog.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dd52a28 <avahi-daemon.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dcf7428 <system-setup-keyboard.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211e68328 <systemd-logind.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211e3eb28 <irqbalance.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020f639b28 <abrt-xorg.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880210800b28 <NetworkManager.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dcff828 <acpid.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d983828 <gpm.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020ff69d28 <mcelog.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d983728 <abrt-oops.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d983d28 <abrtd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211a5ab28 <smartd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880210af7228 <auditd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8802116f8e28 <iscsid.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020e2aa928 <bluetooth.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d943828 <fsck@.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020e95ba28 <udev.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8802116f8228 <systemd-journald.service>
             [no tasks queued]
       [120] PID: 27130  TASK: ffff8801c2c3ae20  COMMAND: "kworker/1:2"
       GROUP CFS RB_ROOT: ffff8801ed860028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ed860828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020dab6e28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ef2d4e28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ef1ed928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8800a07d7f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ef2aca28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f11ff228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f180b828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020ee24f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e111428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880210a5fd28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e160028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020f502528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f5b528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ff7d28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020ddf7728
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880210a5e828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e114e28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f5c028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8802129c2c28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8d6a828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f7d3f028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f822f028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8fbce28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8fbd028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f822eb28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8d76e28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f58628
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8e7c728
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8fbd828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f59528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8e7e228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020ee78928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020d983028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ed5d28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9b93928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9b03f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020edd5f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9a71e28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e2aad28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020dcfe328
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ec8728
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020edd4628
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020d925328
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211e69628
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020dd58028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211f26a28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211e68f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ef0c28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ef1328
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ef1028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020d972728
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e2aaa28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020db0c228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020db0c728
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020a844328
          [no tasks queued]

  CPU 2
    CURRENT: PID: 0      TASK: ffff880212958000  COMMAND: "swapper/2"
    RT PRIO_ARRAY: ffff88021e293e28
       [100] GROUP RT PRIO_ARRAY: ffff88020db22800 <system>
             [100] GROUP RT PRIO_ARRAY: ffff8801f7ee7000 <udisks2.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8e10000 <colord-sane.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8802116f7000 <colord.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f1fb800 <cups.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f53800 <upower.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f994b800 <accounts-daemon.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880212ae4000 <prefdm.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8181800 <atd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9abf000 <crond.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9abb000 <autofs.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f56000 <sm-client.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f7ee5000 <sendmail.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f7e21000 <nscd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f7d01800 <wpa_supplicant.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f69000 <ypbind.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f6e800 <nfs-lock.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8e13800 <rpcbind.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8d7b800 <libvirtd.service>
                   [100] GROUP RT PRIO_ARRAY: ffff8801f8079000 <libvirt>
                         [100] GROUP RT PRIO_ARRAY: ffff8801f7e23800 <lxc>
                               [no tasks queued]
                         [100] GROUP RT PRIO_ARRAY: ffff88021091a000 <qemu>
                               [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020dc47000 <sshd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211f49800 <ksmtuned.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9a7d000 <dbus.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9a79000 <rsyslog.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f99ce800 <avahi-daemon.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f99ca800
<system-setup-keyboard.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9976800 <systemd-logind.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801fc2cd800 <irqbalance.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f98c7800 <abrt-xorg.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9886800 <NetworkManager.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020ee38800 <acpid.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020ee3c800 <gpm.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f40c800 <mcelog.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211ed8800 <abrt-oops.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f401800 <abrtd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f405000 <smartd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801fc2c8800 <auditd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211a55800 <iscsid.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020eee2000 <bluetooth.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020e634000 <fsck@.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211edd000 <udev.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880212add000 <systemd-journald.service>
                   [no tasks queued]
    CFS RB_ROOT: ffff880215806068
       GROUP CFS RB_ROOT: ffff8801ed861928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ed861c28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020dab6928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ef2d5028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ef1edf28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8800a07d6428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ef2ada28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f11fe428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f180b128
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020ee25728
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e110728
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880210a5f928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e160b28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e114428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f5b028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ff7e28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f822ff28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880210a5f628
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e115f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f5c828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8802129c2028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8d6a928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f7d3f928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f822f428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8fbcf28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8fbd228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f822e928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8d76328
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f58428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8e7ca28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8fbc128
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f59628
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8e7e328
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9a73228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801fc3e8628
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ed5c28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9b93128
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9b03b28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211a5a528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9a71f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020f02e528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020dcfeb28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ec9428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020edd5828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020d924c28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211e68228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020d9f3d28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211f27928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211e68928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e111e28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ef0f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ef1d28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020d973228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e2ab128
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020db0d028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020db0d628
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ee1628 <system>
          GROUP CFS RB_ROOT: ffff88020ed3fb28 <udisks2.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f814cb28 <colord-sane.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dd0fc28 <colord.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88021041a228 <cups.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dd99628 <upower.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880210aaba28 <accounts-daemon.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f7f6df28 <prefdm.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f7f6d428 <atd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d00628 <crond.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d00428 <autofs.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d981b28 <sm-client.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8f5c328 <sendmail.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d77a28 <nscd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8e99428 <wpa_supplicant.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d6bd28 <ypbind.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d943528 <nfs-lock.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801fc232328 <rpcbind.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d980b28 <libvirtd.service>
             GROUP CFS RB_ROOT: ffff8801f82b6328 <libvirt>
                GROUP CFS RB_ROOT: ffff8801f828de28 <lxc>
                   [no tasks queued]
                GROUP CFS RB_ROOT: ffff8801f82b6928 <qemu>
                   [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801fc233128 <sshd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f9a70728 <ksmtuned.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020ddf6528 <dbus.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f9a77928 <rsyslog.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dd53028 <avahi-daemon.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dcf6e28 <system-setup-keyboard.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211e3f528 <systemd-logind.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211e3e428 <irqbalance.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020f638128 <abrt-xorg.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880210800028 <NetworkManager.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dcfec28 <acpid.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d982828 <gpm.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020ff69028 <mcelog.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d982c28 <abrt-oops.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d982728 <abrtd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211a5a228 <smartd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8802116f8f28 <auditd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8802116f9728 <iscsid.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020e2ab328 <bluetooth.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d942528 <fsck@.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8802116f8a28 <udev.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8802116f8528 <systemd-journald.service>
             [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020a844f28
          [no tasks queued]

  CPU 3
    CURRENT: PID: 4733   TASK: ffff8801eed74530  COMMAND: "crash"
    RT PRIO_ARRAY: ffff88021e2d3e28
       [100] GROUP RT PRIO_ARRAY: ffff88020db23000 <system>
             [100] GROUP RT PRIO_ARRAY: ffff8801f7ee0000 <udisks2.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8e17000 <colord-sane.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8802116f1000 <colord.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f1fa000 <cups.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f53000 <upower.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f994f800 <accounts-daemon.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880212ae4800 <prefdm.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8182000 <atd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9abf800 <crond.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9abb800 <autofs.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f56800 <sm-client.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f7ee4800 <sendmail.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f7e20000 <nscd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f7d02000 <wpa_supplicant.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f68800 <ypbind.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8f6f000 <nfs-lock.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8e14000 <rpcbind.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f8d7c000 <libvirtd.service>
                   [100] GROUP RT PRIO_ARRAY: ffff880211ec1000 <libvirt>
                         [100] GROUP RT PRIO_ARRAY: ffff8801f7e24000 <lxc>
                               [no tasks queued]
                         [100] GROUP RT PRIO_ARRAY: ffff88021091c800 <qemu>
                               [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8802108ab000 <sshd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211f4e800 <ksmtuned.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9a7d800 <dbus.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9a79800 <rsyslog.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f99cf000 <avahi-daemon.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f99cb000
<system-setup-keyboard.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9977000 <systemd-logind.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801fc2ce000 <irqbalance.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9948000 <abrt-xorg.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801f9887000 <NetworkManager.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020ee38000 <acpid.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020ee3c000 <gpm.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f408000 <mcelog.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211ed8000 <abrt-oops.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f403800 <abrtd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020f405800 <smartd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff8801fc2c9000 <auditd.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211a53800 <iscsid.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020eee1000 <bluetooth.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff88020e632800 <fsck@.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880211edd800 <udev.service>
                   [no tasks queued]
             [100] GROUP RT PRIO_ARRAY: ffff880212add800 <systemd-journald.service>
                   [no tasks queued]
    CFS RB_ROOT: ffff880215806068
       GROUP CFS RB_ROOT: ffff8801ed860b28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ef2b7b28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020dcff028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ef2d4528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ef1ecf28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8800a07d7b28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801ef2ad528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f11ff828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f180a128
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020ee24928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e111028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880210a5fa28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e161a28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9a73928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f5ae28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ff7f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f822fe28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880210a5e428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e115b28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f5d228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8802129c2228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8d6b128
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f7d3ed28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f822f628
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8fbc028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8fbd128
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f822e828
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8d76228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8e7d228
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8e7cb28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8fbc428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8f59728
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f8e7e428
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020edd4a28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801fc3e8b28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ed5628
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9b92f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9b03e28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211a5bb28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff8801f9a71528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020f02ef28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020dcfea28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ec9728
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020edd5e28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020d925e28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211e69f28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ed5928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211f26528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211e69d28
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e111928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ef1628
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020f75e728
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020d972528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020e2aa028
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020db0d528
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020db0c928
          [no tasks queued]
       GROUP CFS RB_ROOT: ffff880211ee1f28 <system>
          GROUP CFS RB_ROOT: ffff88020ed3fc28 <udisks2.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020f52de28 <colord-sane.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dd0fe28 <colord.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020f006128 <cups.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dd99728 <upower.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880210aab228 <accounts-daemon.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f7f6c328 <prefdm.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f7f6d028 <atd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d00228 <crond.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d01828 <autofs.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020f638c28 <sm-client.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8f5cc28 <sendmail.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d76928 <nscd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8e99128 <wpa_supplicant.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f8d6be28 <ypbind.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020e152b28 <nfs-lock.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801fc233b28 <rpcbind.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d980428 <libvirtd.service>
             GROUP CFS RB_ROOT: ffff8801f82b6428 <libvirt>
                GROUP CFS RB_ROOT: ffff8801f828c928 <lxc>
                   [no tasks queued]
                GROUP CFS RB_ROOT: ffff8801f82b6b28 <qemu>
                   [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801fc233328 <sshd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f9a70c28 <ksmtuned.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020ddf7a28 <dbus.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8801f9a77a28 <rsyslog.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dd53b28 <avahi-daemon.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dcf7628 <system-setup-keyboard.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211e3fb28 <systemd-logind.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211e3f428 <irqbalance.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020f638528 <abrt-xorg.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880210801428 <NetworkManager.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020dcff128 <acpid.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d983e28 <gpm.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020ff68628 <mcelog.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d982a28 <abrt-oops.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d982128 <abrtd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880211a5a028 <smartd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020de77828 <auditd.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff880210af7a28 <iscsid.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020e2ab828 <bluetooth.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff88020d943d28 <fsck@.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8802116f9128 <udev.service>
             [no tasks queued]
          GROUP CFS RB_ROOT: ffff8802116f8828 <systemd-journald.service>
             [no tasks queued]
       GROUP CFS RB_ROOT: ffff88020a844128
          [no tasks queued]
  crash> 

In fact, it's difficult to actually find a real *task* that is on
a run queue from among all of the empty "[no tasks queued]" groups!

Dave

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Re: [Crash-utility] [PATCH] runq: make tasks in throttled cfs_rqs/rt_rqs displayed