How do I close pop-up windows with Selenium in Python when I don't know when they will pop up?

I am trying to scrape historical weather data from this website: https://www.worldweatheronline.com/taoyuan-weather-history/tai-wan/tw.aspx

Using this code:

    driver.find_element_by_css_selector("input[type='date']").send_keys(str(for_weather.mm[i])+str(for_weather.dd[i])+for_weather.year[i].astype(str))

wait=WebDriverWait(driver,10)
wait.until(EC.element_to_be_clickable((By.ID,'ctl00_MainContentHolder_butShowPastWeather'))).click()
temp=driver.find_element_by_xpath('//div[@class="days-collapse-temp"]').get_attribute('innerHTML')

One some of the pages, a popup appears.

I've seen help that shows how to choose and close popups, but in my case, we don't know when they will show up. On some pages it appears, some don't. When they do show up, they prevent me from obtaining the data I want, and stops the loop. The following is the error message (it has the characteristics of the popup):

ElementClickInterceptedException: element click intercepted: Element <input type="submit" name="ctl00$MainContentHolder$butShowPastWeather" value="Get Weather" id="ctl00_MainContentHolder_butShowPastWeather" class="btn btn-success ml-2"> is not clickable at point (956, 559). Other element would receive the click: <div class="introjs-overlay" style="inset: 0px; position: fixed; cursor: pointer;"></div>
  (Session info: chrome=97.0.4692.71)

Thanks much!

Solution 1:

Here's a Python Selenium solution that uses SeleniumBase:

First pip install seleniumbase, then copy the example below into a Python file, eg weather_test.py. Then run it with pytest:

pytest weather_test.py --block-ads

from seleniumbase import BaseCase

class MyTestClass(BaseCase):
    def test_base(self):
        self.open("https://www.worldweatheronline.com/taoyuan-weather-history/tai-wan/tw.aspx")
        self.js_click("#ctl00_MainContentHolder_butShowPastWeather")
        for i in range(1, 32):
            date_string = "2021-12-%s" % i
            self.set_attribute("#datePicker input", "value", date_string)
            self.js_click('input[value="Get Weather"]')
            temp = self.get_attribute("div.days-collapse-temp", "innerHTML")
            temp = temp.split("</span>")[-1]
            print("%s : %s" % (date_string, temp))

That will get you the weather in that city for all days in December:

2021-12-1 : 11°/13°c
2021-12-2 : 11°/13°c
2021-12-3 : 11°/13°c
2021-12-4 : 11°/13°c
2021-12-5 : 11°/13°c
2021-12-6 : 11°/13°c
2021-12-7 : 11°/13°c
2021-12-8 : 11°/13°c
2021-12-9 : 11°/13°c
2021-12-10 : 17°/21°c
2021-12-11 : 17°/25°c
2021-12-12 : 17°/20°c
2021-12-13 : 14°/15°c
2021-12-14 : 15°/23°c
2021-12-15 : 15°/28°c
2021-12-16 : 17°/27°c
2021-12-17 : 12°/18°c
2021-12-18 : 11°/13°c
2021-12-19 : 12°/18°c
2021-12-20 : 15°/18°c
2021-12-21 : 16°/18°c
2021-12-22 : 17°/18°c
2021-12-23 : 16°/19°c
2021-12-24 : 16°/21°c
2021-12-25 : 13°/14°c
2021-12-26 : 9°/12°c
2021-12-27 : 9°/11°c
2021-12-28 : 10°/18°c
2021-12-29 : 14°/17°c
2021-12-30 : 12°/13°c
2021-12-31 : 11°/14°c

The key difference is that it uses js_click to click on things instead of a regular click, which would give you a ElementClickInterceptedException if there was a pop-up.