Apache 2 UserDir for only one VirtualHost

I only removed the UserDir public.www (in my case UserDir public_html) from the userdir.conf. Then I added that line to my site configurations where I wanted mod_userdir to be active.

<VirtualHost *:80>
        ServerName userdirs.example.com

        <IfModule mod_userdir.c>
                UserDir public_html
        </IfModule>
</VirtualHost>