Why I can't scrape this website even though the html content is visible?
This is the link to scrape
I don't know why but I can't scrape the data after the which contains the entire data in this website. I used requests_html and beautiful soup but there was no successful results :( I just gave a basic code.
url = "https://www.bcorporation.net/en-us/find-a-b-corp/search?refinement=countries%3DUnited%20Kingdom&refinement=countries%3DAustralia&refinement=countries%3DBelgium&refinement=countries%3DCanada&refinement=countries%3DChina&refinement=countries%3DDenmark&refinement=countries%3DFrance&refinement=countries%3DGermany&refinement=countries%3DIreland&refinement=countries%3DItaly&refinement=countries%3DNetherlands%20The&refinement=countries%3DMexico&refinement=countries%3DNew%20Zealand&refinement=countries%3DPortugal&refinement=countries%3DSpain&refinement=countries%3DSweden&refinement=countries%3DSwitzerland&refinement=countries%3DUnited%20States"
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246 '
}
session = HTMLSession()
r = session.get(url, headers=headers)
r.html.render(sleep=50, timeout=50)
content = r.html.find("ul")
title = r.html.find("div[data-testid = company_name]")
print(title)
looking at the Page the data-testid is company-name not company_name :)
title = r.html.find("div[data-testid=company-name]")
worked for me