Does Django scale? [closed]

Solution 1:

1. "What are the largest sites built on Django today?"

   There isn't any single place that collects information about traffic on Django built sites, so I'll have to take a stab at it using data from various locations. First, we have a list of Django sites on the front page of the main Django project page and then a list of Django built sites at djangosites.org. Going through the lists and picking some that I know have decent traffic we see:

   • Instagram: What Powers Instagram: Hundreds of Instances, Dozens of Technologies.

   • Pinterest: Alexa rank 37 (21.4.2015) and 70 Million users in 2013

   • Bitbucket: 200TB of Code and 2.500.000 Users

   • Disqus: Serving 400 million people with Python.

   • curse.com: 600k daily visits.

   • tabblo.com: 44k daily visits, see Ned Batchelder's posts Infrastructure for modern web sites.

   • chesspark.com: Alexa rank about 179k.

   • pownce.com (no longer active): alexa rank about 65k. Mike Malone of Pownce, in his EuroDjangoCon presentation on Scaling Django Web Apps says "hundreds of hits per second". This is a very good presentation on how to scale Django, and makes some good points including (current) shortcomings in Django scalability.

   • HP had a site built with Django 1.5: ePrint center. However, as for novemer/2015 the entire website was migrated and this link is just a redirect. This website was a world-wide service attending subscription to Instant Ink and related services HP offered (*).

2. "Can Django deal with 100,000 users daily, each visiting the site for a couple of hours?"

   Yes, see above.

3. "Could a site like Stack Overflow run on Django?"

   My gut feeling is yes but, as others answered and Mike Malone mentions in his presentation, database design is critical. Strong proof might also be found at www.cnprog.com if we can find any reliable traffic stats. Anyway, it's not just something that will happen by throwing together a bunch of Django models :)

There are, of course, many more sites and bloggers of interest, but I have got to stop somewhere!

Blog post about Using Django to build high-traffic site michaelmoore.com described as a top 10,000 website. Quantcast stats and compete.com stats.

_{(*) The author of the edit, including such reference, used to work as outsourced developer in that project.}

Solution 2:

We're doing load testing now. We think we can support 240 concurrent requests (a sustained rate of 120 hits per second 24x7) without any significant degradation in the server performance. That would be 432,000 hits per hour. Response times aren't small (our transactions are large) but there's no degradation from our baseline performance as the load increases.

We're using Apache front-ending Django and MySQL. The OS is Red Hat Enterprise Linux (RHEL). 64-bit. We use mod_wsgi in daemon mode for Django. We've done no cache or database optimization other than to accept the defaults.

We're all in one VM on a 64-bit Dell with (I think) 32Gb RAM.

Since performance is almost the same for 20 or 200 concurrent users, we don't need to spend huge amounts of time "tweaking". Instead we simply need to keep our base performance up through ordinary SSL performance improvements, ordinary database design and implementation (indexing, etc.), ordinary firewall performance improvements, etc.

What we do measure is our load test laptops struggling under the insane workload of 15 processes running 16 concurrent threads of requests.

Solution 3:

Not sure about the number of daily visits but here are a few examples of large Django sites:

• disqus.com (talk from djangocon)
• bitbucket.org (write up)
• lanyrd.com (source)
• support.mozilla.com (source code)
• addons.mozilla.org (source code) (talk from djangocon)
• theonion.com (write up)
• The guardian.co.uk comment system uses Django (source)
• instagram
• pinterest
• rdio

Here is a link to list of high traffic Django sites on Quora.

Solution 4:

What's the "largest" site that's built on Django today? (I measure size mostly by user traffic)

In the US, it was Mahalo. I'm told they handle roughly 10 million uniques a month. Now, in 2019, Mahalo is powered by Ruby on Rails.

Abroad, the Globo network (a network of news, sports, and entertainment sites in Brazil); Alexa ranks them in to top 100 globally (around 80th currently).

Other notable Django users include PBS, National Geographic, Discovery, NASA (actually a number of different divisions within NASA), and the Library of Congress.

Can Django deal with 100k users daily, each visiting the site for a couple of hours?

Yes -- but only if you've written your application right, and if you've got enough hardware. Django's not a magic bullet.

Could a site like StackOverflow run on Django?

Yes (but see above).

Technology-wise, easily: see soclone for one attempt. Traffic-wise, compete pegs StackOverflow at under 1 million uniques per month. I can name at least dozen Django sites with more traffic than SO.

Solution 5:

Scaling Web apps is not about web frameworks or languages, is about your architecture. It's about how you handle you browser cache, your database cache, how you use non-standard persistence providers (like CouchDB), how tuned is your database and a lot of other stuff...