Web Scraping With Haskell

Solution 1:

http://hackage.haskell.org/package/shpider

Shpider is a web automation library for Haskell. It allows you to quickly write crawlers, and for simple cases ( like following links ) even without reading the page source.

It has useful features such as turning relative links from a page into absolute links, options to authorize transactions only on a given domain, and the option to only download html documents.

It also provides a nice syntax for filling out forms.

An example:

 runShpider $ do
      download "http://apage.com"
      theForm : _ <- getFormsByAction "http://anotherpage.com"
      sendForm $ fillOutForm theForm $ pairs $ do
            "occupation" =: "unemployed Haskell programmer"
            "location" =: "mother's house"

(Edit in 2018 -- shpider is deprecated, these days https://hackage.haskell.org/package/scalpel might be a good replacement)

Solution 2:

From my searching on the Haskell mailing lists, it appears that TagSoup is the dominant choice for parsing pages. For example: http://www.haskell.org/pipermail/haskell-cafe/2008-August/045721.html

As far as the other aspects of web scraping (such as crawling, spidering, and caching), I searched http://hackage.haskell.org/package/ for those keywords but didn't find anything promising. I even skimmed through packages mentioning "http" but nothing jumped out at me.

Note: I'm not a regular Haskeller, so I hope others can chime in if I missed something.

Solution 3:

Although I'm still for now a beginner in Haskell, I have the strong opinion that HTML parsing in 2012 must be done using CSS selectors, and it seems the libraries recommended so far don't use that principle.

One possibility is HandsomeSoup, which is built on top of HXT:

http://egonschiele.github.com/HandsomeSoup/

http://codingtales.com/2012/04/25/scraping-html-with-handsomesoup-in-haskell

This page about HXT, on which HandsomeSoup relies, will also be helpful (you're going to need getText or deep getText):

http://adit.io/posts/2012-04-14-working_with_HTML_in_haskell.html

But another choice is dom-selector:

http://hackage.haskell.org/package/dom-selector

It is right now alpha and its long-term maintenance could be a problem. The advantage of dom-selector is that I couldn't get unicode characters to work with HandsomeSoup. They worked out of the box with dom-selector.

This question is related to that: Is it possible to use Text or ByteString on HXT in Haskell?

dom-selector is based on html-conduit and xml-conduit, for which maintenance appears assured.

EDIT: note my newer answer about lens-based parsing. I left this answer as it's still good on its own, but I would now personally rather use the other approach.

Solution 4:

I wrote another answer to this question already, suggesting CSS selectors-based parsing, however that answer is now a year and a half old, and nowadays I think lenses might be a better approach in haskell. In effect you get something like type-safe compiled selectors.

See this reddit discussion for a couple of options in that vein. In case the link disappears, I copy the direct links:

• taggy-lens
• hexpat-lens
• xml-lens, which apparently can be used with html-conduit

I have used none of those yet, but if I would write new code parsing HTML today, I would definitely go with a lens-based approach.