How to deal with the captcha when doing Web Scraping in Puppeteer?

Solution 1:

This is a reCAPTCHA (version 2, check out demos here), which is shown to you as the owner of the page does not want you to automatically crawl the page.

Your options are the following:

Option 1: Stop crawling or try to use an official API

As the owner of the page does not want you to crawl that page, you could simply respect that decision and stop crawling. Maybe there is a documented API that you can use.

Option 2: Automate/Outsource the captcha solving

There is an entire industry which has people (often in developing countries) filling out captchas for other people's bots. I will not link to any particular site, but you can check out the other answer from Md. Abu Taher for more information on the topic or search for captcha solver.

Option 3: Solve the captcha yourself

For this, let me explain how reCAPTCHA works and what happens when you visit a page using it.

How reCAPTCHA (v2) works

Each page has an ID, which you can check by looking at the source code, example:

<div class="g-recaptcha form-field" data-sitekey="ID_OF_THE_WEBSITE_LONG_RANDOM_STRING"></div>

When the reCAPTCHA code is loaded it will add a response textarea to the form with no value. It will look like this:

<textarea id="g-recaptcha-response" name="g-recaptcha-response" class="g-recaptcha-response" style="... display: none;"></textarea>

After you solved the challenge, reCAPTCHA will add a very long string to this text field (which can then later be checked by the server/reCAPTCHA service in the backend) when the form is submitted.

How to solve the captcha yourself

By copying the value of the textarea field you can transfer the "solved challenge" from one browser to another (this is also what the solving services to for you). The full process looks like this:

Detect if the page uses reCAPTCHA (e.g. check for .g-recaptcha) in the "crawling" browser
Open a second browser in non-headless mode with the same URL
Solve the captcha yourself
Read the value from: document.querySelector('#g-recaptcha-response').value
Put that value into the first browser: document.querySelector('#g-recaptcha-response').value = '...'
Submit the form

Further information/reading

There is not much public information from Google how exactly reCAPTCHA works as this is a cat-and-mouse game between bot creators and Google detection algorithms, but there are some resources online with more information:

Official docs from Google: Obviously, they just explain the basics and not how it works "in the back"
InsideReCaptcha: This is a project from 2014 which tries to "reverse-engineer" reCAPTCHA. Although this is quite old, there is still a lot of useful information on the page.
Another question on stackoverflow: This question contains some useful information about reCAPTCHA, but also many speculative (and very likely) outdated approaches on how to fool a reCAPTCHA.

Solution 2:

You should use combination of following:

Use an API if the target website provides that. It's the most legal way.
Increase wait time between scraping request, do not send mass request to the server.
Change/rotate IP frequently.
Change user agent, browser viewport size and fingerprint.
Use third party solutions for captcha.
Resolve the captcha by yourself, check the answer by Thomas Dondorf. Basically you need to wait for the captcha to appear on another browser, solve it from there. Third party solutions does this for you.

Disclaimer: Do not use anti-captcha plugins/services to misuse resources. Resources are expensive.

Basically the idea is to use anti-captcha services like (2captcha) to deal with persisting recaptcha.

You can use this plugin called puppeteer-extra-plugin-recaptcha by berstend.

// puppeteer-extra is a drop-in replacement for puppeteer,
// it augments the installed puppeteer with plugin functionality
const puppeteer = require('puppeteer-extra')

// add recaptcha plugin and provide it your 2captcha token
// 2captcha is the builtin solution provider but others work as well.
const RecaptchaPlugin = require('puppeteer-extra-plugin-recaptcha')
puppeteer.use(
  RecaptchaPlugin({
    provider: { id: '2captcha', token: 'XXXXXXX' },
    visualFeedback: true // colorize reCAPTCHAs (violet = detected, green = solved)
  })
)

Afterwards you can run the browser as usual. It will pick up any captcha on the page and attempt to resolve it. You have to find the submit button which varies from site to site if it exists.

// puppeteer usage as normal
puppeteer.launch({ headless: true }).then(async browser => {
  const page = await browser.newPage()
  await page.goto('https://www.google.com/recaptcha/api2/demo')

  // That's it, a single line of code to solve reCAPTCHAs 🎉
  await page.solveRecaptchas()

  await Promise.all([
    page.waitForNavigation(),
    page.click(`#recaptcha-demo-submit`)
  ])
  await page.screenshot({ path: 'response.png', fullPage: true })
  await browser.close()
})

PS:

There are other plugins, even I made a very simple one because captcha is getting harder to solve even for a human like me. You can read the code here.
I am strongly not affiliated with 2Captcha or any other third party services mentioned above.
I had created my own solution which is similar to the other answer by Thomas Dondorf, but gave up soon since Captcha is getting more ridiculous and I do not have mental energy to resolve them.