Opinions on hosting servers in-house
Solution 1:
I just got done moving our public facing servers to Rackspace Cloud Servers. About a year ago I did what you're thinking about doing because I wanted complete control over my servers, and am a little bit sour on leasing cheap servers (the typical $100 / month "server" that's just a PC).
I finally gave up on the in-house hosting gig because:
- Local bandwidth is either very expensive or not high quality or both. Good quality routers are also very expensive. So those times when a customer was downloading some huge file at the same time Google and Yahoo etc were crawling were not real nice.
- Having hardware in-house means that I have to babysit it. I've already got our local servers to worry about; adding more didn't help me out much.
- Power outages, some jerkwad with a shovel digging in the wrong spot, etc are suddenly a problem. Having local servers go offline is something we always have to worry about, but why should our public servers go offline with them? There have been times when we've had our lines go down for >24 hours. For in-house workers this is manageable, but having our public sites go away for that long can be bad news.
- Disaster recovery planning is more complicated. What do you do if the server goes up in smoke? Do you have another one? How fast can you bring it or some other machine online? If your connection goes down and your phone/cable company says it'll be up "in a day or two", can you wait it out or can you throw your sites up online somewhere quickly? How do you get them there if all of your connections are down? I know these sorts of issues have to be addressed no matter what, but what kind of resources do you have available locally?
The one thing I did like, as you also mentioned, was being able to shoot files up to the web servers very quickly (we do a lot of WebDAV). But the way I figure it...better us having to wait for uploads than our customers having to wait for downloads.
Anyhow, the Rackspace Cloud Server solution addressed just about all of the concerns I've always had with leasing dedicated machines or signing up with a VPS provider (including cost). It offers a lot of those little tricks that real virtualization promises. Anyhow, I won't advertise for them. An alternative might be Amazon EC2.
Long story longer, I sleep better knowing that I'm not going to get a call because our webserver is on fire or that the power company dug up the DSL lines again. I'll let some other chump be responsible for that... in a place where they've actually got the resources at hand to handle these things.
So my suggestion is to keep your public stuff in a good datacenter of some sort. Use those extra servers for in-house tasks. There's usually something you can use them for... testing, special projects, backup, etc.
Solution 2:
Bandwidth. Right now we have 16d/2u through Comcast. [...] require bonding 2+ cable lines.
I don't know if your ISP has any special tricks up his sleeve, but in general you cannot bond / merge multiple consumer-type lines. Your lines would terminate in different IP addresses at your premises, and you can't "bond" IP addresses. You could put half your servers on line 1, and the other half on line 2 -- but that has obvious drawbacks in case of a failure.
Next up, consumer-class lines generally have higher latency (Round Trip Time), and this impacts TCP/IP performance. For large file downloads or very simple webpages with a minimum of objects it's probably not a big deal. For VoIP, more interactive or more performance-critical uses it could be significant.
And lastly, there are benefits to having a "multi-homed network", i.e. a Autonomous System with multiple transit (carrier) providers. You'll enjoy better route diversity, and have better chances of all your customers being able to resolve a route to you at any given time.
In general, my recommendation would be to co-locate your old servers at a friendly datacenter not too far away. You will still benefit from your existing hardware, and your servers will have nice, low-latency, professional tubes to the Internet.
Solution 3:
Obviously, do what you think you need to do, but my opinion would be a resounding "no way". As a small company, you need to focus on more important things than managing server(s) - do what you do best, develop software and sell software - and let someone else with a lot more experience take care of mundane tasks like installing patches to the O/S, backing up servers etc. and battling DOS attacks.
You will never be able to come close to having the power redundancy, A/C capacity, bandwidth capacity and knowledge know how that a big company like Rackspace can provide you 24 hours a day, 7 days a week, 365 days a year for a few hundred a month.
I ran my own servers for years - moved them all over to the Rackspace cloud - and have never looked back. Now I develop software, and someone else takes care of the infrastructure.
I have to admit I liked the thought of having the server sitting here right next to me, but the reality is they didn't need to be.