Apache - listen only on specific domain, not IP

How can I configure apache so that it refuses connections coming directly to the IP address (http://xxx.xxx.xxx.xxx) instead of the vhost name http://example.com?

My VirtualHost configuration:

ServerName example.com

<VirtualHost *:80>

        ServerName example.com

        DocumentRoot /var/www/           
        <Directory /var/www/>                    
                AllowOverride All
                Order allow,deny
                allow from all
        </Directory>

</VirtualHost>

You cannot have it refuse connections, since the hostname (or IP) that the user is trying to use as their HTTP host is not known to the server until the client actually sends an HTTP request. The TCP listener is always bound to the IP address.

Would an HTTP error response be acceptable instead?

<VirtualHost *:80>
    ServerName catchall
    <Location />
        Order allow,deny
        Deny from all
    </Location>
</VirtualHost>

<VirtualHost *:80>
    ServerName example.com
    DocumentRoot /var/www/
    <Directory /var/www/>
        AllowOverride All
        Order allow,deny
        allow from all
    </Directory>
</VirtualHost>

<VirtualHost *:80>

ServerName xxx.xxx.xxx.xxx  #your server web server IP
        <Location />
                Order deny,allow
                Deny from all
        </Location>

</VirtualHost>

This is will Forbidden access for accessing by IP

A clean way to handle this is with a RewriteRule as follows

<If "%{HTTP_HOST} == 'x.x.x.x'">
  RewriteRule ^.*$ http://www.example.com/$1 [L]
</If>

Though this does expose the site associated with the ip versus denying access to a server called by ip. But simple and clean if this is acceptable.