Best practices to serve a large number of files

Currently we have one Windows Server 2008 R2 as a domain controller and file server.

We're serving about 6 million JPGs to 4 IIS servers which are then served to web users.

The problem is we're getting memory alerts on the server daily. Our current server serves these files from a single drive (LUN). We also have an issue with the number of files on the disc. The master file table is going to grow too large as we continue to add more images.

Our current plan is to:

• move the file server off to a separate VM
• add 4 or 5 separate drives (LUNs) to serve these files
• move to around 15 million images by the end of this year (a little more than double our current number)

My questions are:

1. What is best practice for serving files?
2. Are we ok moving to a single server with 4 or 5 Drives (LUNs) or should we move to 2 file servers with 3 drives each?

Solution 1:

How are you getting memory alerts? Have you done any process analysis to see if the memory usage is somehow out of the norm? Remember that LSASS and the cache will grow to the point of physical memory.

UPDATE:

Not getting alerts under Windows Server 2003 doesn't surprise me since it was not as efficient at using RAM as Windows Server 2008 is. You want as much memory used as possible in use server, which is why monitoring just memory usage alone is a useless metric. Starting with Windows Vista, Windows uses as much RAM as possible. You can see this same behavior on Windows 7 by bringing up the resource monitor and looking at the memory tab. Here is an image from my machine currently:

Notice I have only 64MB free! That's a good thing. (it used to be 2MB but I must have just closed something). Notice that the blue bar shows 1369MB of standby RAM - that's essentially the cache in the filesystem and if I needed more RAM in an application, that's where the pages come from.

So what you need to know is not how much RAM is used but how much each application is using and whether or not that particular app's memory usage is normal.

A good way of determining that (for generating a warning alert) would be to look at memory usage and page faults per second. If you have excessive page faults AND you have high ram utilization AND you have low standby RAM, it's likely there is an issue. (on a fileserver page faults are not as reliable an indicator of problems since it's serving files - thus generating page faults, you must baseline this before setting a threshold)

Hopefully this brief overview on memory monitoring helps.

Solution 2:

You should move to 2 (or more) or more servers for redundancy. If you load balance them, it will be easier to take one down/reboot for maintenance. If you have 3 servers, but only really need 2, then your maintenace window could be during the day, as well.

I would also spread out the drives (LUNs) across these servers - again for redundancy.