How to scrape SVG element from a website using Beautiful Soup?

Solution 1:

The webpage is rendered dynamically with Javascript, so you will need selenium to get the rendered page.

First, install the libraries

pip install selenium
pip install webdriver-manager

Then, you can use it to access the full page

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

s=Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=s)
driver.maximize_window()
driver.get('https://codeforces.com/profile/akash77')
elements = driver.find_elements(By.XPATH, '//*[@id="userActivityGraph"]')

Elements is a selenium WebElement, so we will need to get HTML out of it.

svg = [WebElement.get_attribute('innerHTML') for WebElement in elements]

This gives you svg and all elements inside it.

enter image description here

Sometimes, you need to run a browser in headless mode (without opening a chrome UI), for that you can pass a 'headless' option to the driver.

from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('headless')

# then pass options to the driver

driver = webdriver.Chrome(service=s, options=options)