How can I parse a website using Selenium and Beautifulsoup in python? [closed]
New to programming and figured out how to navigate to where I need to go using Selenium. I'd like to parse the data now but not sure where to start. Can someone hold my hand a sec and point me in the right direction?
Any help appreciated -
Assuming you are on the page you want to parse, Selenium stores the source HTML in the driver's page_source
attribute. You would then load the page_source
into BeautifulSoup
as follows:
In [8]: from bs4 import BeautifulSoup
In [9]: from selenium import webdriver
In [10]: driver = webdriver.Firefox()
In [11]: driver.get('http://news.ycombinator.com')
In [12]: html = driver.page_source
In [13]: soup = BeautifulSoup(html)
In [14]: for tag in soup.find_all('title'):
....: print tag.text
....:
....:
Hacker News
As your question isn't particularly concrete, here's a simple example. To do something more useful read the BS docs. You will also find plenty of examples of selenium (and BS )usage here in SO.
from selenium import webdriver
from bs4 import BeautifulSoup
browser=webdriver.Firefox()
browser.get('http://webpage.com')
soup=BeautifulSoup(browser.page_source)
#do something useful
#prints all the links with corresponding text
for link in soup.find_all('a'):
print link.get('href',None),link.get_text()