Apache custom log filter on Ubuntu using SetEnvIf

Posted under » Apache » LAMP Security on 13 January 2016

Sometimes we suffer from information overload looking at the logs. You want to declutter.

First, your Apache must load the SetEnvIf module which on Ubuntu is already loaded by default.

2nd. please note that whatever changes you have made, it will only come into effect after you restart your Apache.

3rd. Don't edit the log file as root, eg delete some lines on top. Once you edit, log can't write to the log. Best to just delete the file. This is because sometimes, the owner of the log is not root. It could be adm or www-data Restart Apache and it will work as it should.

Where you put these directive is up to you. Some people prefer to implement it server-wide by putting it at the apache2.conf BUT it is best to put it on the /etc/apache2/sites-available where the respective VirtualHosts configuration resides. It may look like this.

   SetEnvIf Remote_Addr "221\.136\.65\.7" dontlog
   SetEnvIf User-Agent ".*internal dummy connection.*" dontlog

	ErrorLog ${APACHE_LOG_DIR}/error-localhost.log
   LogLevel warn
	CustomLog ${APACHE_LOG_DIR}/access-localhost.log combined env=!dontlog

Before we go into details of the above config, let's look at a typical access log.

123.126.113.87 - - [13/Jan/2016:11:14:07 +0800] "GET / HTTP/1.1" 200 576 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
120.153.23.151 - - [10/Jan/2016:05:04:11 +0800] "GET /images/lkyshit.jpg HTTP/1.1" 200 9497 "https://www.google.ca/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"

Customlog

This is a "combined" log. This is by default config
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

%h The remote host (IP)
%l The remote logname (usually just "-")
%u The authenticated user (if any, usually just "-")
%t The time of the access
\"%r\" The first line of the request
%>s The final status of the request
%b The size of the server's response, in bytes
\"%{Referer}i\" The referrer URL, taken from the request's headers
\"%{User-Agent}i\" The user agent, taken from the request's headers

You can make your own log format and give it a nickname. Combined is itself an alias/nickname which you can modify.

LogFormat "%h %l %u %t \"%r\" %>s %b  %T" whuteva

CustomLog /var/log/apache2/whateva.log whuteva

So you can have several custom logs of the same website.

SetEnvIf

With SetEnvIf, you can prevent or allow requests from getting logged based on the following criteria:

Host
User-Agent
Referer
Accept-Language
Remote_Host: the hostname (if available) of the client making the request.
Remote_Addr: the IP address of the client making the request.
Server_Addr: the IP address of the server on which the request was received.
Request_Method: the name of the method being used (GET, POST, etc.).
Request_Protocol: the name and version of the protocol with which the request was made (e.g., "HTTP/0.9", "HTTP/1.1", etc.).
Request_URI: the resource requested on the HTTP request line - generally the portion of the URL following the scheme and host portion without the query string.

The SetEnvIf directive has the following form: SetEnvIf attribute regex env-variable

SetEnvIf attribute regex [!]env-variable[=value] [[!]env-variable[=value]]

To log gif access

SetEnvIf Request_URI "\.gif$" object_is_image=gif

To prevent jpeg from NOT being logged:

SetEnvIf Request_URI "\.gif\.jpg$" !object_is_image=gif

To prevent all requests made with a certain browser, e.g. Internet Explorer, from getting logged, you could use:

SetEnvIf User_Agent "(MSIE)" dontlog

To not log requests from any client whose hostname ends in heh.example.com, use:

SetEnvIf Remote_Host "heh.example.com$" dontlog

To not log requests from any client whose hostname begins with example, use:

SetEnvIf Remote_Host "^example" dontlog

To not log requests from a certain IP address, use something like:

SetEnvIf Remote_Addr "192\.168\.0\.154" dontlog

If you don't want requests of your robots.txt to get logged, use:

SetEnvIf Request_URI "^/robots\.txt$" dontlog

To not log lky access. Note that is like REGEX.

SetEnvIf Request_URI "^/lky/sucks$" dontlog

Apart from SetEnvIf, which is case-sensitive, you can use SetEnvIfNoCase which is case-insensitive. For example, in order not to log certain search engine spiders, you could use:

SetEnvIFNoCase User-Agent "Googlebot" dontlog

Or to not log certain file extensions, use something like this:

SetEnvIfNoCase Request_URI "\.(gif)|(jpg)|(png)|(css)|(js)|(ico)$" dontlog

To not log certain referrals (e.g. from your own domain), use something like:

SetEnvIfNoCase Referer "www\.anoneh\.com" dontlog

Putting it all together

   SetEnvIf Remote_Addr "221\.136\.65\.7" dontlog
   LogFormat "%h %l %u %t \"%r\" %>s %b  %T" whuteva

   ErrorLog ${APACHE_LOG_DIR}/error-localhost.log
   LogLevel warn

	CustomLog ${APACHE_LOG_DIR}/access-localhost.log combined
	CustomLog ${APACHE_LOG_DIR}/whuteva-localhost.log whuteva env=!dontlog
	CustomLog ${APACHE_LOG_DIR}/NOTwhuteva-localhost.log whuteva env=dontlog

Please note the difference of env=!dontlog and env=dontlog. If you don't put the format name 'whuteva' then there will be no format.. eg. IP address etc.

You can parse custom Apache log data as a messaging service.

I find tail -f handy when I want to see the log in real time.

 tail -f /var/log/apache2/access-localhost.log