Deny requests for other domains in Apache 2

Let's say I have xy.com running on a server powered by Apache. It's the default virtual host.
Now someone has rewritten his hosts file to redirect yz.com to my server's IP. As my server doesn't know a thing about yz.com, it just resolves it to the default virtual host. So in this case yz.com/hello get's resolved to xy.com/hello and it tries to serve my site's /hello page. Is there a way (maybe with a RewriteRule or something) to disable requests requesting yz.com from my server? Give them a 403 for example or anything, just don't let anything through Apache unless it's requesting xy.com.
I'm using the latest Apache version on Ubuntu LTS.

Why is xy.com the default if you don't want it to be?

Add a new default virtual host that does nothing but return a 404.

Then add xy.com as a virtual host entry. It'll only get used if requested.

Why not simply change the default VHost to a deny one?

<VirtualHost *:80>
    ServerName lol.no
    DocumentRoot /var/empty/httpd
    <Location />
        Deny from all
        Allow from none
    </Location>
</VirtualHost>

<VirtualHost *:80>
    ...your VHost goes here...
</VirtualHost>

A RewriteRule would be something like this:

RewriteCond %{HTTP_HOST} !^your\.site\.com$ [NC]
RewriteRule .* - [F]

Other solution would be to add a new default Virtual Host which responds with 403 to any requests.
Only matching requests would then go to your real Virtual Host.