audit.d crashing - Auditd save files consuming large amount of disk space in /var/log/audit.d/save

Today, as I was about to copy over some directories from an old drive on a Linux web server, I was shocked to notice that my SCSI hard disk dive was 90% full. BTW, it is a good system administration practice to always check for disk space before you copy over large directories. To quickly view disk usage statistics in human readable form, use the following command.

df -h

So I was really amazed at how quickly the disk space was becoming full. Right away, I knew I had probably forgotten to stop some logging services on the server. The problem was which services are producing these logs?

I ran disk usage command (du) on the server log directory to see which logs were eating up the disk space. The command I used was

du -h -s /var/log/*

From the output I was able to confirm that the log files creating the trouble were in the directory /var/log/audit.d/. The /var/log/audit.d/ directory was reportedly using 35G of space.

First, lets verify whether auditd (audit service) is still running. On RHEL systems and most modern Linux installations, audit daemon status can be verified by using the following command.

service audit status

The output, if audit daemon is running, will be similar to the following:

auditd (pid 1262) is running...

To stop the auditd service, we can use

service audit stop

To verify audit daemon has infact stopped,

[root@plain skins]# service audit stop
Shutting down audit subsystem [ OK ]
[root@plain skins]# service audit status
auditd is stopped

Another way to verify that auditd is no longer running, we can use ps and grep

[root@plain skins]# ps -ef | grep auditd
root 20874 14450 0 21:44 pts/5 00:00:00 grep auditd

The "-e" option select all processes while the "-f" option for ps command gives full process listing
Note, that the process displayed in the above command output is not the auditd process running but rather the grep process running.

Also note, that in case you were unable to stop auditd processes, you can always use killall command with signal 9 to forcefully kill all running processes for the process name specified. For example,

killall -9 auditd

Next, we will turn off audit daemon. To do this, I executed

chkconfig audit off

No output was generated. According to chkconfig manual, chkconfig [service] off modifies the bootup information for the service specified via command line. From the man entry for chkconfig:

If one of on, off, or reset is specified after the service name, chk-
config changes the startup information for the specified service. The
on and off flags cause the service to be started or stopped, respec-
tively, in the runlevels being changed. The reset flag resets the
startup information for the service to whatever is specified in the
init script in question.

By default, the on and off options affect only runlevels 2, 3, 4, and
5, while reset affects all of the runlevels. The --level option may be
used to specify which runlevels are affected.

Now lets stop crond for a moment

service crond stop

Stop the at daemon (atd) if running.

service atd stop

Before removing the auditd mod (next), if you want, you can use lsmod to see audit mod installed (but then why are you reading this?)

[root@plain skins]# lsmod | grep audit
audit 90744 1

Now, remove the audit mod by using rmmod

rmmod audit

Now if we run the lsmod command, no output will be displayed.

Next edit the file /etc/modules.conf and append the following line.

alias char-major-10-224 off

The above should also solve the modprobe not being able to locate char-major-10-224 issue in case you are noticing the following in your error logs (Source)

modprobe: modprobe: Can't locate module char-major-10-224

Lets start the cron daemon

service crond start

and the at daemon (if you were running it)

service atd start

If you followed everything correctly, the auditd program should now be turned off, and shouldn't start automatically on boot or through a cron daemon. If you have any questions or comments, please add them using the "Add comments" link below. Remember to leave your e-mail (won't be publically visible) if you would like for me to contact you.

Please note, I take no responsibility if something bad happens as a result of you following this tutorial. Use this only as a resource to research further and only if you know what you are doing.


And oh, the following command can be used to remove (be very careful or you may end up deleting wrong files.) the old auditd log files generated.

rm /var/log/audit.d/save.*

Auditd Sun Docs
Tags: linux linux kernel system administration security audit auditd crash
log+files internet server crontab desktop