Apache custom log filter on Ubuntu 14 using SetEnvIf

Posted under » Apache » LAMP Security on 13 January 2016

Sometimes we suffer from information overload looking at the logs. You want to declutter.

First, your Apache must load the SetEnvIf module which on Ubuntu 14 is already loaded by default.

2nd. please note that whatever changes you have made, it will only come into effect after you restart your Apache.

Where you put these directive is up to you. Some people prefer to implement it server-wide by putting it at the apache2.conf BUT it is best to put it on the /etc/apache2/sites-available where the respective VirtualHosts configuration resides. It may look like this.

   SetEnvIf Remote_Addr "221\.136\.65\.7" dontlog
   SetEnvIf User-Agent ".*internal dummy connection.*" dontlog

	ErrorLog ${APACHE_LOG_DIR}/error-localhost.log
   LogLevel warn
	CustomLog ${APACHE_LOG_DIR}/access-localhost.log combined env=!dontlog

Before we go into details of the above config, let's look at a typical access log.

123.126.113.87 - - [13/Jan/2016:11:14:07 +0800] "GET / HTTP/1.1" 200 576 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
120.153.23.151 - - [10/Jan/2016:05:04:11 +0800] "GET /images/lkyshit.jpg HTTP/1.1" 200 9497 "https://www.google.ca/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"

Customlog

This is a "combined" log. This is by default config
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

You can make your own log format and give it a nickname. Combined is itself an alias/nickname which you can modify.

LogFormat "%h %l %u %t \"%r\" %>s %b  %T" whuteva

CustomLog /var/log/apache2/whateva.log whuteva

So you can have several custom logs of the same website.

SetEnvIf

With SetEnvIf, you can prevent or allow requests from getting logged based on the following criteria:

The SetEnvIf directive has the following form: SetEnvIf attribute regex env-variable

SetEnvIf attribute regex [!]env-variable[=value] [[!]env-variable[=value]]

To log gif access

SetEnvIf Request_URI "\.gif$" object_is_image=gif

To prevent jpeg from NOT being logged:

SetEnvIf Request_URI "\.gif\.jpg$" !object_is_image=gif

To prevent all requests made with a certain browser, e.g. Internet Explorer, from getting logged, you could use:

SetEnvIf User_Agent "(MSIE)" dontlog

To not log requests from any client whose hostname ends in heh.example.com, use:

SetEnvIf Remote_Host "heh.example.com$" dontlog

To not log requests from any client whose hostname begins with example, use:

SetEnvIf Remote_Host "^example" dontlog

To not log requests from a certain IP address, use something like:

SetEnvIf Remote_Addr "192\.168\.0\.154" dontlog

If you don't want requests of your robots.txt to get logged, use:

SetEnvIf Request_URI "^/robots\.txt$" dontlog

To not log lky access

SetEnvIf Request_URI "^/lky/sucks$" dontlog

Apart from SetEnvIf, which is case-sensitive, you can use SetEnvIfNoCase which is case-insensitive. For example, in order not to log certain search engine spiders, you could use:

SetEnvIFNoCase User-Agent "Googlebot" dontlog

Or to not log certain file extensions, use something like this:

SetEnvIfNoCase Request_URI "\.(gif)|(jpg)|(png)|(css)|(js)|(ico)$" dontlog
To not log certain referrals (e.g. from your own domain), use something like:
SetEnvIfNoCase Referer "www\.anoneh\.com" dontlog

Putting it all together

   SetEnvIf Remote_Addr "221\.136\.65\.7" dontlog
   LogFormat "%h %l %u %t \"%r\" %>s %b  %T" whuteva

   ErrorLog ${APACHE_LOG_DIR}/error-localhost.log
   LogLevel warn

	CustomLog ${APACHE_LOG_DIR}/access-localhost.log combined
	CustomLog ${APACHE_LOG_DIR}/whuteva-localhost.log whuteva env=!dontlog
	CustomLog ${APACHE_LOG_DIR}/NOTwhuteva-localhost.log whuteva env=dontlog

Please note the difference of env=!dontlog and env=dontlog

I find tail -f handy when I want to see the log in real time.

 tail -f /var/log/apache2/access-localhost.log

 

web security linux ubuntu python django git Raspberry apache mysql php drupal cake javascript css AWS