Virtualization Pitfalls/Lessons Learned

ALWAYS eject any virtual media (CD/DVD/Floppy) once you've finished as failure to do so will often stop a vMotion in its tracks.

Get your NTP and DNS setup correctly, this will save you from contemplating suicide :)

You can never have enough memory, or storage.

Make sure you have remote, OS-less, access to your machines such as HP's iLO system.

Keep a repository of OS/App .ISO files.

Not a direct answer to your question but in the hope that someone saves themselves from tearing their hair out in the future by finding this response - HP blade servers don't ship with their 'VT'-bit enabled by default, you have to enable it in the BIOS (F9). Without this ESX 3.5U4 doesn't throw a useful error, no it just hangs prior to the code install :(

To answer the question as asked - pitfalls related to P2V migrations.

First off - P2V migrations work very well for the most part. The cleaner and newer the systems the better but even with migrating old (NT4 systems) my success rate after more than a hundred migrations in a range of environments has been around 90%. That's systems that have migrated and been handed over to production on the day (well mostly on the night) planned. I've only ever had one system that we had to reverse out of after an apparently successful migration - a SQL box that needed more CPU horsepower than the platform could ever deliver. VMware Converter is good and free (for the non enterprise version), Platespin is very good (but costly).

That said - there are things to avoid.

MSCS Clusters. You can make them work but it's never a great idea and Microsoft will almost certainly not help you in any way if you have problems later. Build new stand alone systems instead.

Large SQL servers - emphasis on large. These should have been red flagged from a CPU requirements POV in advance but don't be tempted to move one if you aren't certain that the target VM will have ample CPU headroom.

If you are planning to change system names or ip-addresses (or both) during the migration then first consider don't doing that and if you absolutely have no choice then make sure you have people on hand who understand how those changes might affect the systems in question. My worst migration ever was an RSA ACE server used for authenticating a DMZ located VPN where the client refused to listen to my objections and insisted on changing both name and ip-address during the migration.

Related to the above - if you have anything other than a completely flat network then build some test VM's and make 100% sure that your VM networks perfectly replicate the physical ones you are migrating from.

In Windows AD environments always make sure you have a local admin account on the box being migrated. And test it before you migrate.

Make sure you have a good idea of how long things will take. P2V copy times will vary depending on available network bandwidth (obviously) but also can be dramatically affected by the number of files in each volume being migrated. This is particularly a problem with Platespin migrating NT4* systems but will affect any P2V software copying at the file level (which generally applies if you opt to resize volumes). Copy rates of 70-80Megabyte a second are possible with GigE networks, relatively fast source and a good target setup but 20-30Megabyte/sec is more typical and for the aforementioned NT systems with 100Meg networks and lots of files I've seen copy rates drop down into the 50kilobyte/sec range.

• Ideally you would get rid of these but some people don't have that luxury and getting such an OS of the completely unrepairable hardware its probably running on is almost always a good idea.

• Have a solid backup strategy in place beforehand. Decide if you are going to back up the VM as if it were on bare metal, or if you are going to back up the virtual hard disks on the data stores (or both). Generally speaking, I found that my required backup footprint increased significantly at first, so be prepared for an initial spike where you might be backing up both an old physical machine and a new VM before you get all of the kinks worked out.
• VM sprawl is also something to watch out for. Once virtualization starts taking off, the urge to move everything to VM becomes great. While this can work, you probably didn't order exactly enough hardware right off the bat.
• I think there are machines that can't be converted, and other machines that probably shouldn't be converted. While it is nice to be able to take a 10 year old physical machine and clone it to a VM, with warts and all, there are certainly scenarios where you would be better off building a
  clean OS and migrating objects from the physical machine over. Sometimes you are better off not converting over the cobwebs.
• Be prepared to use a lot of network ports. If you have systems that run on different VLANS, while single ports could be trunked, you probably want to have individual ports for your VLANs feeding into your vSwitch. If you want redundancy, and you are using iSCSI, you could be looking at a lot of NICs.

From my experience, be VERY careful about your storage medium. We went with an iSCSI SAN that turned out to only support 100Mbit connections. Running one VM on the system wasn't bad, two was less the adequate... and by the time we reached our target of 8 VMs they were horrible.

My personal lesson learned: Check the rated IOPS and read more reviews about a product that relate to the way you intend to use the storage device

Another handy thing I've learned... Making a 'backup' disk image after a base install and hardening will quicken the build of any other system and is a very handy thing to keep around.

Try not to run production database servers in a Virtual Environment. The overheads for I/O is unacceptable. We had huuuge problems when our DBA allowed our primary MSSQL server to become virtualised. Queries were taking thousands of milliseconds to run. When we convinced them to move it back to a dedicated box, there was a 10,000% increase in throughput and speed.