BeautifulSoup not grabbing dynamic content

My issue I'm having is that I want to grab the related links from this page: http://support.apple.com/kb/TS1538

If I Inspect Element in Chrome or Safari I can see the <div id="outer_related_articles"> and all the articles listed. If I attempt to grab it with BeautifulSoup it will grab the page and everything except the related articles.

Here's what I have so far:

import urllib2
from bs4 import BeautifulSoup
url = "http://support.apple.com/kb/TS1538"
response = urllib2.urlopen(url)
soup = BeautifulSoup(response.read())
print soup

Solution 1:

This section is loaded using Javascript. Disable your browser's Javascript to see how BeautifulSoup "sees" the page.

From here you have two options:

• Use a headless browser, that will execute the Javascript. See this questions about this: Headless Browser for Python (Javascript support REQUIRED!)
• Try and figure out how the apple site loads the content and simulate it - it probably does an AJAX call to some address.

After some digging it seems it does a request to this address (http://km.support.apple.com/kb/index?page=kmdata&requestid=2&query=iOS%3A%20Device%20not%20recognized%20in%20iTunes%20for%20Windows&locale=en_US&src=support_site.related_articles.TS1538&excludeids=TS1538&callback=KmLoader.receiveSuccess) and uses JSONP to load the results with KmLoader.receiveSuccess being the name of the receiving function. Use Firebug of Chrome dev tools to inspect the page in more detail.

Solution 2:

I ran into a similar problem, the html contents that are created dynamically may not be captured by BeautifulSoup. A very basic solution for this is to make it wait for few seconds before capturing the contents, or use Selenium instead that has the functionality to wait for an element and then proceed. So for the former, this worked for me:

import time

# .... your initial bs4 code here

time.sleep(5) #5 seconds, it worked with 1 second too
html_source = browser.page_source

# .... do whatever you want to do with bs4