Chrome: API for performance data
Problem Descritpion
How can I access the data from the Chrome Performance tool either from the browser console or using Chrome driver methods when using Selenium?
In particular, I am interested in getting the page loading time. On the top diagrams I can see that the last elements were loaded around 1700-1800 ms mark. The event log in the bottom shows that the last element started loading (and finished in a few ms) at 1720.1 ms from the time the page was reloaded.
What I Tried So Far
I tried using Performance Timing interface like this:
console.log(performance.timing.loadEventEnd -
performance.timing.navigationStart)
357
But I guess it measures some other performance. loadEventEnd and navigationStart have the most extreme values among the attributes of timing and their difference is only 357 (ms).
I also tried logging performance info using Selenium like this (took some ideas from here and rewrote in Python):
import time
import json
import copy
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.chrome.options import Options
caps = copy.deepcopy(DesiredCapabilities.CHROME)
caps['loggingPrefs'] = {'performance': 'ALL'}
chromedriver_options = Options()
chromedriver_options.add_argument("--headless")
chromedriver_options.add_argument("--window-size=1920,1080")
chromedriver_options.add_argument("--log-level=3") # suppress selenium logging to stdout
driver = webdriver.Chrome(chrome_options=chromedriver_options, executable_path='chromedriver.exe', desired_capabilities=caps)
driver.set_window_position(0, 0)
driver.implicitly_wait(8) # set timeout to 8 seconds
driver.get(my_url)
time.sleep(5) # wait for page to load for the first time
_ = driver.get_log('performance') # discard all previous logs
driver.refresh()
time.sleep(5) # make sure page loaded completely before writing log
with open('logs.txt', 'a') as f:
for entry in driver.get_log('performance'):
f.write(str(entry) + '\n')
driver.close()
The resulting logs contain entries with calls to Page.frameStartedLoading and Page.frameStoppedLoading methods and their timestamps and I had an idea to measure the time between the first Page.frameStartedLoading and the last Page.frameStoppedLoading after flushing the log and calling driver.refresh().
Unfortunately, the resulting log file doesn't make a lot of sense. The performance logging doesn't stop (I thought it should since the web page is loaded at some point and nothing is happening performance-wise). If I call time.sleep(5) after refreshing, I would get some logs with 5 calls to each Page.frameStartedLoading and Page.frameStoppedLoading. If I call time.sleep(7) I would get 7 calls to each Page.frameStartedLoading and Page.frameStoppedLoading. And the difference between timestamp of the last Page.frameStoppedLoading and the first Page.frameStartedLoading would be ~7 seconds in this last case which is clearly not what I wanted.
Anyone has experience with capturing this page loading performance info?
Solution 1:
If you want to access the Performance Data from the Chrome Developer Tool using ChromeDriver you can use the following code block :
-
Selenium-Python Snippet
from selenium import webdriver driver=webdriver.Chrome(executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe') driver.get("http://www.google.com") performance_data = driver.execute_script("return window.performance.getEntries();") print (performance_data) -
The parameters extracted are as follows :
connectEnd Time when server connection is finished. connectStart Time just before server connection begins. domComplete Time just before document readiness completes. domContentLoadedEventEnd Time after DOMContentLoaded event completes. domContentLoadedEventStart Time just before DOMContentLoaded starts. domInteractive Time just before readiness set to interactive. domLoading Time just before readiness set to loading. domainLookupEnd Time after domain name lookup. domainLookupStart Time just before domain name lookup. fetchStart Time when the resource starts being fetched. loadEventEnd Time when the load event is complete. loadEventStart Time just before the load event is fired. navigationStart Time after the previous document begins unload. redirectCount Number of redirects since the last non-redirect. redirectEnd Time after last redirect response ends. redirectStart Time of fetch that initiated a redirect. requestStart Time just before a server request. responseEnd Time after the end of a response or connection. responseStart Time just before the start of a response. timing Reference to a performance timing object. navigation Reference to performance navigation object. performance Reference to performance object for a window. type Type of the last non-redirect navigation event. unloadEventEnd Time after the previous document is unloaded. unloadEventStart Time just before the unload event is fired.
Console Output :
Page Title is : Google
[{'connectEnd': 2725.146514015987, 'connectStart': 1503.1734234800108, 'decodedBodySize': 204837, 'domComplete': 7603.4821342458235, 'domContentLoadedEventEnd': 4588.4400826362535, 'domContentLoadedEventStart': 4576.987229310746, 'domInteractive': 4559.208601432438, 'domainLookupEnd': 1503.1734234800108, 'domainLookupStart': 1503.1734234800108, 'duration': 7623.300733238722, 'encodedBodySize': 61075, 'entryType': 'navigation', 'fetchStart': 1503.1734234800108, 'initiatorType': 'navigation', 'loadEventEnd': 7623.300733238722, 'loadEventStart': 7603.516579526861, 'name': 'document', 'nextHopProtocol': 'h2', 'redirectCount': 0, 'redirectEnd': 1501.686197816412, 'redirectStart': 22.057947498907886, 'requestStart': 2725.729247123413, 'responseEnd': 3624.7713441197043, 'responseStart': 3349.3679493549, 'secureConnectionStart': 2049.7916668355165, 'startTime': 0, 'transferSize': 61926, 'type': 'navigate', 'unloadEventEnd': 0, 'unloadEventStart': 0, 'workerStart': 0}, {'connectEnd': 3388.3040845619494, 'connectStart': 3388.3040845619494, 'decodedBodySize': 1730, 'domainLookupEnd': 3388.3040845619494, 'domainLookupStart': 3388.3040845619494, 'duration': 237.3373068328902, 'encodedBodySize': 1730, 'entryType': 'resource', 'fetchStart': 3388.3040845619494, 'initiatorType': 'img', 'name': 'https://www.google.co.in/logos/doodles/2018/doodle-snow-games-day-15-5907794870927360.4-s.png', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 3439.3892730100556, 'responseEnd': 3625.6413913948395, 'responseStart': 3625.271003314041, 'secureConnectionStart': 0, 'startTime': 3388.3040845619494, 'transferSize': 2242, 'workerStart': 0}, {'connectEnd': 0, 'connectStart': 0, 'decodedBodySize': 0, 'domainLookupEnd': 0, 'domainLookupStart': 0, 'duration': 230.20794413542762, 'encodedBodySize': 0, 'entryType': 'resource', 'fetchStart': 3425.689371645131, 'initiatorType': 'css', 'name': 'https://ssl.gstatic.com/gb/images/i1_1967ca6a.png', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 0, 'responseEnd': 3655.8973157805594, 'responseStart': 0, 'secureConnectionStart': 0, 'startTime': 3425.689371645131, 'transferSize': 0, 'workerStart': 0}, {'connectEnd': 0, 'connectStart': 0, 'decodedBodySize': 0, 'domainLookupEnd': 0, 'domainLookupStart': 0, 'duration': 969.9734406621285, 'encodedBodySize': 0, 'entryType': 'resource', 'fetchStart': 3502.3913129811713, 'initiatorType': 'script', 'name': 'https://www.gstatic.com/external_hosted/createjs/createjs-2015.11.26.min.js', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 0, 'responseEnd': 4472.364753643298, 'responseStart': 0, 'secureConnectionStart': 0, 'startTime': 3502.3913129811713, 'transferSize': 0, 'workerStart': 0}, {'connectEnd': 3520.930168473896, 'connectStart': 3520.930168473896, 'decodedBodySize': 12232, 'domainLookupEnd': 3520.930168473896, 'domainLookupStart': 3520.930168473896, 'duration': 134.0411771046674, 'encodedBodySize': 12232, 'entryType': 'resource', 'fetchStart': 3520.930168473896, 'initiatorType': 'css', 'name': 'https://www.google.co.in/logos/2018/snowgames_bobsled/cta.png', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 3525.3821197382813, 'responseEnd': 3654.971345578564, 'responseStart': 3625.955856548778, 'secureConnectionStart': 0, 'startTime': 3520.930168473896, 'transferSize': 12745, 'workerStart': 0}, {'connectEnd': 4527.024551785848, 'connectStart': 4527.024551785848, 'decodedBodySize': 842405, 'domainLookupEnd': 4527.024551785848, 'domainLookupStart': 4527.024551785848, 'duration': 2632.3253968704244, 'encodedBodySize': 265174, 'entryType': 'resource', 'fetchStart': 4527.024551785848, 'initiatorType': 'script', 'name': 'https://www.google.co.in/logos/2018/snowgames_bobsled/snowgames_bobsled18.2.js', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 4528.843667863202, 'responseEnd': 7159.3499486562705, 'responseStart': 4946.163646693538, 'secureConnectionStart': 0, 'startTime': 4527.024551785848, 'transferSize': 265741, 'workerStart': 0}, {'connectEnd': 4558.896162471504, 'connectStart': 4558.896162471504, 'decodedBodySize': 0, 'domainLookupEnd': 4558.896162471504, 'domainLookupStart': 4558.896162471504, 'duration': 1005.3106518587082, 'encodedBodySize': 0, 'entryType': 'resource', 'fetchStart': 4558.896162471504, 'initiatorType': 'other', 'name': 'https://www.google.co.in/gen_204?s=webaft&atyp=csi&ei=N6mPWp6mC8H00ASJtoL4Ag&rt=wsrt.3384,aft.1175,prt.1175', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 4560.575876470308, 'responseEnd': 5564.206814330209, 'responseStart': 5564.1444075857435, 'secureConnectionStart': 0, 'startTime': 4558.896162471504, 'transferSize': 358, 'workerStart': 0}, {'connectEnd': 4576.353841378264, 'connectStart': 4576.353841378264, 'decodedBodySize': 417606, 'domainLookupEnd': 4576.353841378264, 'domainLookupStart': 4576.353841378264, 'duration': 1903.6302084304214, 'encodedBodySize': 144417, 'entryType': 'resource', 'fetchStart': 4576.353841378264, 'initiatorType': 'script', 'name': 'https://www.google.co.in/xjs/_/js/k=xjs.s.en.smQ6-n1iGHA.O/m=sx,sb,cdos,cr,elog,hsm,jsa,r,d,csi/am=wCLkeMEAyP8JgogEKwgsQIpgGBA/rt=j/d=1/t=zcms/rs=ACT90oH8FZej9QmzW2qBqlOOQ7DASmnKAA', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 4578.110550711152, 'responseEnd': 6479.984049808687, 'responseStart': 5564.372556917789, 'secureConnectionStart': 0, 'startTime': 4576.353841378264, 'transferSize': 144999, 'workerStart': 0}, {'connectEnd': 0, 'connectStart': 0, 'decodedBodySize': 0, 'domainLookupEnd': 0, 'domainLookupStart': 0, 'duration': 574.9679253647753, 'encodedBodySize': 0, 'entryType': 'resource', 'fetchStart': 4791.30009458269, 'initiatorType': 'script', 'name': 'https://www.gstatic.com/og/_/js/k=og.og2.en_US.SpCLDXmWlPM.O/rt=j/m=def/exm=in,fot/d=1/ed=1/rs=AA2YrTtMoJJMGQfOfYZyZ7reaiaiva99OQ', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 0, 'responseEnd': 5366.2680199474635, 'responseStart': 0, 'secureConnectionStart': 0, 'startTime': 4791.30009458269, 'transferSize': 0, 'workerStart': 0}, {'connectEnd': 0, 'connectStart': 0, 'decodedBodySize': 0, 'domainLookupEnd': 0, 'domainLookupStart': 0, 'duration': 1586.0147296126488, 'encodedBodySize': 0, 'entryType': 'resource', 'fetchStart': 5417.5655534222005, 'initiatorType': 'script', 'name': 'https://apis.google.com/_/scs/abc-static/_/js/k=gapi.gapi.en.29tAKSAI8cc.O/m=gapi_iframes,googleapis_client,plusone/rt=j/sv=1/d=1/ed=1/am=IA/rs=AHpOoo82FxkTgGRAoVn-fgFU3zdQ5QIqEw/cb=gapi.loaded_0', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 0, 'responseEnd': 7003.580283034848, 'responseStart': 0, 'secureConnectionStart': 0, 'startTime': 5417.5655534222005, 'transferSize': 0, 'workerStart': 0}, {'connectEnd': 6646.626267079796, 'connectStart': 6646.626267079796, 'decodedBodySize': 83168, 'domainLookupEnd': 6646.626267079796, 'domainLookupStart': 6646.626267079796, 'duration': 783.8056119062338, 'encodedBodySize': 27299, 'entryType': 'resource', 'fetchStart': 6646.626267079796, 'initiatorType': 'script', 'name': 'https://www.google.co.in/xjs/_/js/k=xjs.s.en.smQ6-n1iGHA.O/m=aa,abd,async,dvl,foot,fpe,ipv6,lu,m,mu,sf,sonic,d3l/am=wCLkeMEAyP8JgogEKwgsQIpgGBA/exm=sx,sb,cdos,cr,elog,hsm,jsa,r,d,csi/rt=j/d=1/ed=1/t=zcms/rs=ACT90oH8FZej9QmzW2qBqlOOQ7DASmnKAA?xjs=s1', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 6648.111871788991, 'responseEnd': 7430.431878986027, 'responseStart': 7159.731278179281, 'secureConnectionStart': 0, 'startTime': 6646.626267079796, 'transferSize': 27880, 'workerStart': 0}, {'connectEnd': 0, 'connectStart': 0, 'decodedBodySize': 0, 'domainLookupEnd': 0, 'domainLookupStart': 0, 'duration': 930.0310980101976, 'encodedBodySize': 0, 'entryType': 'resource', 'fetchStart': 6668.677730761095, 'initiatorType': 'img', 'name': 'https://www.google.com/textinputassistant/tia.png', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 0, 'responseEnd': 7598.708828771292, 'responseStart': 0, 'secureConnectionStart': 0, 'startTime': 6668.677730761095, 'transferSize': 0, 'workerStart': 0}, {'connectEnd': 7222.302954756732, 'connectStart': 7222.302954756732, 'decodedBodySize': 469, 'domainLookupEnd': 7222.302954756732, 'domainLookupStart': 7222.302954756732, 'duration': 208.68612730489625, 'encodedBodySize': 469, 'entryType': 'resource', 'fetchStart': 7222.302954756732, 'initiatorType': 'img', 'name': 'https://www.google.co.in/logos/2018/snowgames_bobsled/main-sprite.png', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 7223.911346761861, 'responseEnd': 7430.989082061627, 'responseStart': 7430.7698479787905, 'secureConnectionStart': 0, 'startTime': 7222.302954756732, 'transferSize': 980, 'workerStart': 0}]
From this output you can easily pull out the performance stats of your interest.
Solution 2:
Right-click on a Performance recording and select Save Profile. It'll give you a LOT of data, but should probably include whatever you need (it's the source data for the Performance recording report visualization, I think).
The underlying protocol that DevTools uses to get data from the browser is called the Chrome DevTools Protocol. Other clients, such as Selenium, can start a session with the browser (like DevTools does), and use the protocol to get the same data that DevTools gets. The underlying protocol method is called Performance.getMetrics. I'm not familiar with Selenium, but it should be possible to hook into the CDP, somehow.
You can also automate this with Puppeteer. I provide an example of using Puppeteer to access DevTools features in Using DevTools Features Without Opening DevTools.