Is there an advantage to load balancing using multiple VMs all on the same hardware?

The way I see it is this: With the virtualization, you sandbox some of the processing resources. You separate out into discrete units the web servers; you can upgrade a server while having another to take over, and can redirect requests if there's a problem.

You can snapshot the VM's to roll back from production problems.

You can more easily migrate the VM's to separate hardware when/if you get it, so instead of migrating bare metal setups (configure a whole new web server) you can just move the VM to a new machine, or if you set up a "proper" cluster your VM's will be a in a good position to just roll into production rather than reconfigure from scratch.

You can set up separate VSwitches to the NICs to control traffic flow, if need be.

I think it provides some greater flexibility and options down the road to have multiple machines on the VM host handling your website.

On the other hand, if you have only one host, you can have problems with point of failure. If you have at least two NICs this helps mitigate it (you can dedicate the NIC to a particular VM, depending on your needs and setup, for example) and depending on your host's configuration some problems will be more pronounced. For example, if your disk volume isn't optimal, you're going to suffer reduced performance because of I/O contention, and your processor of course is going to have to split resources among the VM's.

Really the strength for me in virtualizing this is the easier management and migration path down the road. I think it's a lot easier to abstract the hardware from the machine, then when something better/bigger/faster is purchased, you can migrate your established working setup to that hardware with less downtime and configuration hassle. Or you can balance the machines between hosts. And in our case, if we have a problem there is still a possibility, if we absolutely had to do it run the servers as VM's on workstations until repair parts came in for our host servers using the hypervisor in a temporary setup, without the users really noticing a difference.

And snapshotting (in our use case) has been great for making sure updates don't turn our systems unto paperweights on reboot. Snapshot, update (or alter the configuration), reboot, test, and delete the snap if all goes well.

So does it make sense? If your site is heavily trafficked, maybe not, because you'll have I/O contention. If your site is lesser loaded and you think you'll get another host in the future, I'd strongly consider it. I think the abstraction from virtualization helps make it easier to add machines and alter the configuration over time, but if you're hurting for performance that overhead that virtualizing incurs can hurt a little.

You will incur more overhead than running only one instance. The only benefit will be that if one VM goes down you will have another ready to act as a failover. Your reliability would be better if you had more hardware, of course.

One possible advantage of this is that later on you could move the VMs onto other hardware and subscribe more resources, but you could get the same benefit by just adding more servers to the load balancing as you add capacity.