Possible Issue With (what I think is) a Virtual Network
This really isnt my area of expertise...and it may be obvious, but here it is: We have what I guess you could call a virtual network. We have 4 locations in 4 cities connected by broadband internet (Comcast and Verizon) all 5mb business lines. One location houses a file server and a database server for all other locations. This system has been in place since before my time. We have budgeted for a fiber connection in the next couple of years.
The problem is that opening even the smallest files (mostly Office docs) takes forever. Even files that are just kilobytes in size seem to take a very long time to open or even transfer over. Saving changes to a med-sized Excel (or any Office doc)over the network temp freezes up your machine.
What can I use to troubleshoot this problem? ALso, this issue is worse at certain times of the day and particular bad at a certain location. I should also mention that machines located inside the LAN where the servers are stored have no issues at all.
Am I expecting too much by wanting near-lan-like speeds from this setup?
We have lots in common. One of the things we don't have in common is that you've got a 5Mb/s link, where as I'm using a 1.544Mb/s T1 between my various offices. Here's how I've coped.
First, Robert Moir's suggestions are all spot on and his observations are correct. If I can flesh out one of his points, if you aren't monitoring the traffic on the various network links, do it. MRTG is not the best solution, but it's the simplest, and it makes it easy to see when your peak hours are. It may be (and in my case certainly is) that your slow traffic speeds are caused by your users. My wires run hot pretty much the whole time my users are doing anything. They know that when people are doing large DB queries, transferring files, and downloading content, then the network will be slow.
Since it sounds like you might be using leased lines and not internet-pointed T1s or DSL links like I am, you can take advantage of a device known as a WAN accelerator which gets installed on both ends of a line (one each at your home office and your branch office, for example) and compresses the traffic, which yields an effective higher throughput than the bandwidth is capable of.
Next, make any services local that you possibly can. The less you have to send over the link, the better. If the users have roaming profiles or home directories, keep those at the branch office, or at least sync them nightly so they can use the local copy.
Decentralize your network security. I don't know how your network traffic works, but lots of places make their branch office only have one network connection, to the central office, and so internet bound traffic goes through the same pipe as internal traffic, in order to hit the corporate firewall. That setup is more secure, but it kills your bandwidth. Decentralizing that by getting internet access lines at the offices and installing smaller firewalls on those sounds like the opposite of what every security professional recommends (and it is), but business operations come before security in the IT version of Maslow's hierarchy.
I'm British rather than American so I'm not aware of the typical expectations from Verizon and Comcast's business internet business, but if the problem is worse at different times of the day I'd suspect contention. Do you have a SLA for the performance of these lines and have you verified that it's being met?
That aside you clearly can't expect "near LAN" performance from something that's nowhere near a modern LAN in terms of specification for much the same reason you can't expect "near spaceship" performance from a paper airplane: no matter how good your paper airplane making skills are you just can't get there from here.
To some degree the latency you're likely to see over WAN links is likely to make performance noticably worse than a LAN running at the same speed at the best of times, and you're probably comparing a 5Mb internet connection to a 100Mb or 1000Mb LAN, so clearly there has to be limits on what you expect.
Which isn't to say that the current situation is acceptable either. You need to do a couple of things IMHO, to make the best of the current links:
In the order I'd do this...
Check what's currently going over these network connections to ensure that something unexpected isn't sucking up the bandwidth you do have.
Consider 'caching' documents that don't change often but which are used by all offices (e.g. manuals, policy documents, etc) on "local" servers for each office, even if the "local server" is a nominated PC that is always on.
Take a look at the routers you are using in each office - are they designed for the sort of use you are making of them?
Consider making sure material that does have to travel over the link is served up in a manner friendly to slower links (e.g. place documents on an intranet rather than a file share and it'll probably work better because HTTP is better designed for slower speed connections than windows file sharing, for example)
Definitely speak to your ISPs about the SLA for these lines, contention issues.
Hmm.. Should have added - check for simple things related to name resolution and other things that can slow windows server performance down at the best of times. Maybe experiment with adding hosts file entries for the server(s) to a remote workstation to see if that helps.
To improve file transfers, look at using protocols like SCP which aren't assuming that you're on a LAN.
Also, dig into the specifics of how your networks work and optimize. Windows defaults to a 65k TCP Window size (pre-Vista), which is often too large for WAN connections and fragments your IP packets, thus reducing performance. Tuning this down significantly may improve overall throughput. Ditto with MTU size.
Another thing to try is disabling SNP on Windows servers. SNP offloads work to NIC cards and performs other "optimizations" that typically cause problems.
I work in a big enterprise environment, where we've struggled to deliver services with DSL/Cable and 98% of our remote connectivity is delivered by T-1, MPLS, Frame Relay or Ethernet, so I don't have alot of advice specific to your situation. I'd suggest asking for advice on forums on sites like BroadbandReports and Ars Technica's networking forum for more advice.
Yeah, 5Mbit over any sort of Internet link isn't going to give you reasonable performance for a Windows file share. The latency will absolutely kill you.