need advice on building a scalable architecture for moodle

I'm looking into designing an architecture for a moodle based education site, it will serve several thousand users at first but needs to be able to grow to support hundreds of thousands to millions of users across several countries.

I was thinking of a load balancer to distribute requests to several web servers. The web servers can be split by some serving static and some serving dynamic content. Then it is to write to a mysql master node and read from slave nodes.

What kind of load balancer will work well with moodle, should I get a hardware load balancer solution from one of the vendors, or build one myself with open source solution like LVS or reverse proxy?

I was planning to use apache server to serve the web pages at first then as loads become higher, split into lighttpd webserver for static content and apache application server for dynamic content. Stuff like gzip compressing, squid cache, memcache will also be deployed if required.

For the web server hardware, should I use one-u single socket server or a blade solution? Which one will end up being cheaper to run and expand? Supermicro have an interesting product with twin servers in 1u chassis and 4 servers in 2U chassis with infiniband. Has anyone here tried these server before?

For the storage, should I use a SAN or storage server like Sun unified storage 7000 will be sufficient. For a mysql cluster setup should I have two different storage systems, on to use for master node writing access and another for slave to read? Or should all the nodes have separate storage?

Since this website will likely be more heavy on read operations, what consideration should be made for the mysql cluster and storage setup?

For the management part I am planning to use dsh, ganglia, nagios, splunk, kickstart.

For backup I am planning for an LTO tape autoloader. This site will be primarily used for the Asia region, so there will be several hours of low traffic at night. What is the best way to back up up a mysql cluster? Can I temporarily disable write and take out the master to perform the backup?

Please advise if you have experience with setting up this kind of scalable web site, most of my experience has been in working with large unix boxes, or smaller standalone unix/linux boxes. So this kind of scale out implementation is a first time for me.

Thanks

Robert.

Robert, you're clearly a smart guy, but respectfully, get a consultant with prior domain knowledge, or begin building something small now and see where it takes you. There is no way to answer your post; it has too many abstract concepts and no hard numbers.

A few thoughts:

will serve several thousand users at first ... grow to support hundreds of thousands to millions of users

Prove that you need that level of scale first. Don't build a scale-out architecture in anticipation of users that never show up. Sorry if I sound harsh, but 99% of all websites don't grow to the large end of the scale. See Stack Overflow / Server Fault; they're serving a million users monthly from a handful of fairly conventional servers.

should I get a hardware load balancer solution from one of the vendors, or build one myself with open source solution

Depends on your skills and your situation regarding time vs money. Once built, the open source and commercial offerings work pretty much exactly the same. Commercial solutions tend to have better statistics and nicer management interfaces out of the box.

For the web server hardware, should I use one-u single socket server or a blade solution?

Ask your server vendor for prices. Ask your datacenter about power density, i.e. their preferred balance between size and power consumption -- often you'll be power limited, so a dense solution like blades may not win you anything.

For the storage, should I use a SAN or storage server like Sun unified storage 7000 will be sufficient.

Get SAN when you have a proven need for SAN; then you will also better understand what needs your SAN should solve for you.

Since this website will likely be more heavy on read operations, what consideration should be made for the mysql cluster and storage setup?

Create a really good caching solution. Either full page caching like Squid (Varnish), or application data caching like Memcached, or a combination of both. Consider cache invalidation, could you need to quickly purge content from your caches to avoid it being served again?

What is the best way to back up up a mysql cluster?

Opinions vary, but one common approach is to have a dedicated slave MySQL just for backups, and use something like InnoBackup or Maatkit for a self-scripted backup solution.

Edit: If you're really going to build this from scratch now, then please take a good look at cloud computing before committing. Cloud computing isn't just about scalability, even if scalability is a great strenght. Certain services that come as part of the package can really help in making day to day operations easier. Some examples:

• Live snapshots of Amazon EBS volumes make for easy database backups.
• Amazon has load balancing as a set and forget service (of course more feature limited than good self-hosted load balancer, but easy to get started with).
• Rightscale has extensive server monitoring built into their images, which makes for easy capacity planning / application introspection.

While I don't know much about the specifics of Moodle, I can offer some tips for general scalability.

Blades and SANs are often mis-sold by vendors. I suspect that a cluster of commodity 1U servers would probably be the best for your needs. There's a number of datacentres that won't take blade systems because the power draw is very high, and the cooling requirements are quite needy too!

I'm a big fan of Gluster for distributed/replicated storage, you might find it interesting to investigate as an alternative to a SAN solution from a large vendor.

A whole stack of HP DL360s would do just as well (or cheaper commodity servers (I highly recommend DNUK)). I seriously doubt that you'd need Infiniband interconnections between your servers (the infrastructure is expensive, and largely unnecessary for web serving purposes, if you were doing HPC modelling of genome expression, my answer might be different!)

With regard to network infrastructure (if you've got to consider this too..), I recommend Cisco routers, with either Cisco Catalyst switches, or HP Procurves (pretty evenly matched, IMO, and cheaper)

As far as load-balancing is concerned, a dedicated linux server running LVS will easily handle traffic to multiple cluster nodes. If you had the money ($30k+), then a citrix netscaler might be the right caching/acceleration/load-balancing platform, but bear in mind that you'd need 2 (ideally 3) of them for redundancy.

You should probably try to include memcache from the outset, it's easy to add scalability, and vastly improves caching performance, especially when reading from a MySQL database cluster. There's other things you can do to tune your MySQL performance too, such as using InnoDB over MyISAM.

I suspect you would be better off with a reverse proxy cache, such as Varnish, as opposed to Squid, which works better as a client-side cache.
You could quite easily have a couple of dedicated Varnish cache nodes, or run Varnish on the same server as the Apache / lighttpd servers.

Try to avoid getting into a state where you get vendor lock-in, as this can prove very expensive when it comes to licensing issues. It's very possible to build a scalable site using free/open-source software, entirely. Of course, software load balancers aren't going to be as fast as hardware ones with dedicated ASICs, but with a good network infrastructure, it can come pretty close.

For the management part I am planning to use dsh, ganglia, nagios, splunk, kickstart.

Just need to add puppet to that list, and you're onto a winner. Watch out for splunk's expensive licensing (when you get into processing 10s of GB of logs per day, it might bite you).

Munin is a great free monitoring tool, and has advantages over apps like Zabbix, because it can automatically configure graphs from the plugin script (so you don't have to keep track in advance of what you're monitoring).

While I have never administered a Moodle system that could be considered large (at most one having a few thousand active users) and I'm almost certain you have more Linux experience than me, I can offer some observations.

A Moodle installation with millions of users would be an order of magnitude larger than any others that I have heard of. Even the Open University, with students spread across the UK and the world, is only anticipating 200,000 users. Large US universities tend to have only tens of thousands of users. For a good idea of size have a look at http://docs.moodle.org/en/Large_installations Will you really get millions of people using the system? Are they all going to be appearing at once or gradually enrolling over years? You don't need a system capable of handling millions if you're only going to get 10,000 in the first year. In addition, a lot of institutions have theoretical numbers of students who will use Moodle, but only a small percentage of those actually use the system. In short, start small and scale up.

A Linux distro will make life easiest when administering Moodle. The online community help that's available isn't orientated to Windows at all!

Consider a recruiting a local Moodle Partner (sorry, I'm new and so can't post this second link) if you're serious about this. They can offer advice based upon personal experience in setting-up and administering Moodle systems.

Moodle is generally very easy on system resources. Just watch the databases, as the transaction rate can be very large. You haven't mentioned it, but consider separating db servers from the web server and concentrate resources on clustering the db. With caching (eaccelerator or memcached) web-access is negligible. File storage is also generally non-intensive and a link to a decent raid array, local or on a separate machine, is all that is needed. If you've got a SAN, use it. If not, just stick with simple stuff.

As always, backup, backup, backup!

Good luck!