scrape NBA table without id or class
I am trying to scrape this NBA all time leader data with BeautifulSoup: https://www.nba.com/stats/alltime-leaders/?SeasonType=Regular%20Season&PerMode=Totals&StatCategory=PTS
How can I scrape all the players? I tried this but it did not work.
players = soup.find_all("td", class_="player")
I also tried to scrape by table but it did not have id or class:
enter image description here
Data is loaded dynamically from another endpoint you can find in the network tab of the browser. Calling that endpoint gives a json response you can reconstruct the table from easily:
import requests
import pandas as pd
r = requests.get('https://stats.nba.com/stats/leagueLeaders?ActiveFlag=No&LeagueID=00&PerMode=Totals&Scope=S&Season=All+Time&SeasonType=Regular+Season&StatCategory=PTS').json()
df = pd.DataFrame(r['resultSet']['rowSet'], columns = r['resultSet']['headers'])
print(df)