Re: [Crash-utility] feature to dump audit logs in vmcore

Dave, I wrote an extension module to dump audit logs in vmcore. How about this in crash utility as a built-in command? crash> extend /root/repos/crash-dumpaudit-command/src/dumpaudit.so /root/repos/crash-dumpaudit-command/src/dumpaudit.so: shared object loaded crash> dumpaudit type=1300 audit(1489022639.875:164489): arch=c000003e syscall=0 success=yes exit=0 a0=5 a1=7fedd3b00000 a2=400 a3=22 items=0 ppid=2575 pid=10428 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=1 comm="pidof" exe="/usr/sbin/killall5" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null) type=1320 audit(1489022639.875:164489): type=1320 audit(1489022639.875:164487): type=1300 audit(1489022639.875:164490): arch=c000003e syscall=3 success=yes exit=0 a0=5 a1=1 a2=8 a3=0 items=0 ppid=2575 pid=10428 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=1 comm="pidof" exe="/usr/sbin /killall5" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null) ...<cut>...

What the extension module does is so simple that it retrives audit logs from a queue. Looking back into kernel git logs, this design that audit logs are held in the queue, was introduced at the following patch to introduce kauditd kernel thread and have never changed until now: # git describe b7d1125817c9a46cc46f57db89d9c195e7af22f8 v2.6.12-rc4-48-gb7d1125 # git log -1 -p b7d1125817c9a46cc46f57db89d9c195e7af22f8 commit b7d1125817c9a46cc46f57db89d9c195e7af22f8 Author: David Woodhouse <dwmw2(a)shinybook.infradead.org&gt; Date: Thu May 19 10:56:58 2005 +0100 AUDIT: Send netlink messages from a separate kernel thread netlink_unicast() will attempt to reallocate and will free messages if the socket's rcvbuf limit is reached unless we give it an infinite timeout. So do that, from a kernel thread which is dedicated to spewing stuff up the netlink socket. Signed-off-by: David Woodhouse <dwmw2(a)infradead.org&gt; I confirmed the comamnd works well on fedora 4.8 kernel, RHEL7.3, RHEL6.8 and RHEL5.11; RHEL6.8 was tested both on x86_64 and x86_32. So, I guess the design will rarely be changed also in the future for some time. To get vmcore full of audit logs, run the following commands: # auditctl -a exit,always -S all # always record exit of all system calls # auditctl -e 1 # enable audit logging # auditctl -f 2 # configure kernel to panic when audit buffer becomes full # kill -STOP $(pidof auditd) # stop auditd that is a unique process # to read and reduce audit buffer # dd if=/dev/zero of=/dev/null # issue system calls a lot then kernel will panic when audit buffer becomes full. To dump audit logs, simply, run dumpaudit command: crash> extend /root/repos/crash-dumpaudit-command/src/dumpaudit.so /root/repos/crash-dumpaudit-command/src/dumpaudit.so: shared object loaded crash> dumpaudit type=1300 audit(1489022639.875:164489): arch=c000003e syscall=0 success=yes exit=0 a0=5 a1=7fedd3b00000 a2=400 a3=22 items=0 ppid=2575 pid=10428 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=1 comm="pidof" exe="/usr/sbin/killall5" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null) type=1320 audit(1489022639.875:164489): type=1320 audit(1489022639.875:164487): type=1300 audit(1489022639.875:164490): arch=c000003e syscall=3 success=yes exit=0 a0=5 a1=1 a2=8 a3=0 items=0 ppid=2575 pid=10428 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=1 comm="pidof" exe="/usr/sbin /killall5" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null) ...<cut>... On the vmcore created like above, log command should show the messages ending with: crash> log | tail -n 21 [253655.583070] audit: audit_backlog=321 > audit_backlog_limit=320 [253655.583124] audit: audit_backlog=321 > audit_backlog_limit=320 [253655.583129] audit: audit_lost=1 audit_rate_limit=0 audit_backlog_limit=320 [253655.583139] Kernel panic - not syncing: audit: backlog limit exceeded [253655.583142] CPU: 0 PID: 2575 Comm: bash Not tainted 3.10.0-327.el7.x86_64 #1 [253655.583143] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 [253655.583147] ffffffff81870541 000000007dd10266 ffff8800b8a43d40 ffffffff816351f1 [253655.583149] ffff8800b8a43dc0 ffffffff8162ea6c ffffffff00000010 ffff8800b8a43dd0 [253655.583151] ffff8800b8a43d70 000000007dd10266 0000000000000000 ffffffff818705e8 [253655.583152] Call Trace: [253655.583168] [<ffffffff816351f1>] dump_stack+0x19/0x1b [253655.583170] [<ffffffff8162ea6c>] panic+0xd8/0x1e7 [253655.583178] [<ffffffff81103ce4>] audit_panic+0x64/0x70 [253655.583180] [<ffffffff81103d2f>] audit_log_lost+0x3f/0xd0 [253655.583186] [<ffffffff811049bc>] audit_log_start+0x1bc/0x4b0 [253655.583189] [<ffffffff810919dc>] ? do_send_sig_info+0x6c/0xa0 [253655.583193] [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20 [253655.583199] [<ffffffff81108ce1>] audit_log_exit+0x51/0xb90 [253655.583201] [<ffffffff8110b7cd>] __audit_syscall_exit+0x21d/0x280 [253655.583208] [<ffffffff81645a2b>] sysret_audit+0x17/0x21 where audit_backlog_limit=320 means the limit number of audit logs configured in /etc/audit.conf was 320 and audit_backlog=321 means the audit queue, or audit backlog, became full amounting to 321. dumpaudit should dump audit logs as many lines as the limit: crash> dumpaudit | wc -l 321 -- Thanks. HATAYAMA, Daisuke -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility