JSoup UserAgent, how to set it right?
You might try setting the referrer header as well:
doc = Jsoup.connect("https://www.facebook.com/")
.userAgent("Mozilla/5.0 (Windows; U; WindowsNT 5.1; en-US; rv1.8.1.6) Gecko/20070725 Firefox/2.0.0.6")
.referrer("http://www.google.com")
.get();
Response response= Jsoup.connect(location)
.ignoreContentType(true)
.userAgent("Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0")
.referrer("http://www.google.com")
.timeout(12000)
.followRedirects(true)
.execute();
Document doc = response.parse();
User Agent
Use the latest User agent. Here's the complete list http://www.useragentstring.com/pages/useragentstring.php.
Timeout
Also don't forget to add timout, since sometimes it takes more than normal timeout to download the page.
Referer
Set the referer as google.
Follow redirects
follow redirects to get to the page.
execute() instead of get()
Use execute() to get the Response object. Which can help you to check for content type and status codes incase of error.
Later you can parse the response object to obtain the document.