Apache: Automatic log splitting per Virtual Host?

I feel your pain, as a web developer I have probably over 200 vhosts in my local dev - I personally don't care about the logs & log them all to the main /var/log/apache...

However what I did do was to write a shell script to add/manage all my vhosts - all you need to do is to tweak it to write logs to wherever you like...

#!/bin/bash
ARGS=1
if [ "$1"X = "X" ];
        then
        echo "Must enter domain name"
        exit 0
fi
if [ "$2"X = "X" ];
        then
        echo "Must enter domain suffix"
        exit 0
fi
if [ "$3"X = "X" ];
        then
        echo "you must type "restart" if you want apache restarted "no" if not!"
        exit 0
fi

domain=$1.$2;

#echo $domain;
#exit 0


rm $domain.conf

echo "<VirtualHost *:80>" >> $domain.conf;
echo "        ServerAdmin [email protected]" >> $domain.conf;
echo "        ServerName $1.network.local" >> $domain.conf;
echo "        DocumentRoot /Data/vhome/$1.$2/httpdocs" >> $domain.conf;
echo "        HostnameLookups Off" >> $domain.conf;
echo "        UseCanonicalName Off" >> $domain.conf;
echo "        ServerSignature On" >> $domain.conf;
echo "        ScriptAlias /cgi-bin/ "/Data/vhome/$1.$2/cgi-bin/"" >> $domain.conf;
echo "        ErrorLog /var/log/apache2/error_log" >> $domain.conf;
echo "        CustomLog /var/log/apache2/access_log combined" >> $domain.conf;
echo "    <Directory "/Data/vhome/$1.$2/cgi-bin">" >> $domain.conf;
echo "        AllowOverride All" >> $domain.conf;
echo "        Options +ExecCGI -Includes" >> $domain.conf;
echo "        Order allow,deny" >> $domain.conf;
echo "        Allow from all" >> $domain.conf;
echo "    </Directory>" >> $domain.conf;
echo "    <Directory "/Data/vhome/$1.$2/httpdocs">" >> $domain.conf;
echo "        Options Indexes FollowSymLinks" >> $domain.conf;
echo "        AllowOverride All" >> $domain.conf;
echo "        Order allow,deny" >> $domain.conf;
echo "        Allow from all" >> $domain.conf;
echo "    </Directory>" >> $domain.conf;
echo "       # #XSS prevention" >> $domain.conf;
echo "       # RewriteEngine On" >> $domain.conf;
echo "       # RewriteCond %(REQUEST_METHOD) ^TRACE" >> $domain.conf;
echo "       # RewriteRule .* -[F]" >> $domain.conf;
echo "</VirtualHost>" >> $domain.conf;

if [ "$3" = "restart" ];
        then
        rcapache2 restart;
fi

chmod 666 $domain.conf

cat $domain.conf
echo "Created!";

exit 0

Hope it helps.

-sean

Back when I still used apache, I learned a trick from a coworker. He piped the global access log through an awk script. The awk script in turn would then take care creating the different logfiles.

In httpd.conf:

LogFormat "%V %p %a %l %u %t \"%r\" %s %b \"%200{Referer}i\" \"%200{User-agent}i\" \"%{cookie}n\"" awklogpipe
CustomLog "|/usr/local/bin/apacheawklogpipe" awklogpipe env=!dontlog

The script /usr/local/bin/apacheawklogpipe:

#!/bin/gawk -f

BEGIN {
        months["Jan"] = "01";
        months["Feb"] = "02";
        months["Mar"] = "03";
        months["Apr"] = "04";
        months["May"] = "05";
        months["Jun"] = "06";
        months["Jul"] = "07";
        months["Aug"] = "08";
        months["Sep"] = "09";
        months["Oct"] = "10";
        months["Nov"] = "11";
        months["Dec"] = "12";
}

{
        # HEADS UP: SET THIS!!!!
        LOGBASE="/var/log/httpd/access"

        SSL=""

        # Automagicly set first 
        SITE=tolower($1);
        PORT=$2

        if ($2 == 443)
        {
                SSL="-ssl"
        }

        # Extract all but first two fields (vhostname vhostport to be exactly) into LINE
        LINE=substr($0,index($0,$3));

        # No matter where it is, we will find an apache datestamp.
        match(LINE,/\[[0-9]+\/[A-Z][a-z]+\/[0-9]+:[0-9]+:[0-9]+:[0-9]+[\t ]+[+-][0-9]+\]/);
        split(substr(LINE,RSTART + 1,RLENGTH - 2), ap_log_time, /[\/: ]/);

        #ap_rotatelog= LOGBASE "/" SITE ":" PORT "/access-log" SSL "-" ap_log_time[3] "-" months[ap_log_time[2]] ap_log_time[1];
        ap_rotatelog= LOGBASE "/" SITE  "/access_log" SSL;
        if (system("test -d " LOGBASE "/" SITE ) == 0)
        {
                print LINE >> ap_rotatelog;
                close(ap_rotatelog);
        }
        else
        {
                print SITE "\t" SSL "\t" LINE >> LOGBASE "/w3logrotate-error.log";
                close(LOGBASE "/w3logrotate-error.log");
        }
}

Make sure /usr/local/bin/apacheawklogpipe is executable. All you would need to take care of with this script, is create a directory in /var/log/httpd/access that corresponds to the virtualhostname. I had a script that would create a virtualhost config and create the log directories.

Another options. Don't split them in httpd.conf. Instead, log everything to your main Access Log, and then split them later with a program like split-logfile. This helps to simplify your log configuration.

This is described at http://httpd.apache.org/docs/2.2/logs.html#virtualhost

By adding information on the virtual host to the log format string, it is possible to log all hosts to the same log, and later split the log into individual files. For example, consider the following directives.

LogFormat "%v %l %u %t \"%r\" %>s %b" comonvhost CustomLog
logs/access_log comonvhost

The %v is used to log the name of the virtual host that is serving the request. Then a program like split-logfile can be used to post-process the access log in order to split it into one file per virtual host.