Access Denied page with headless Chrome on Linux while headed Chrome works on windows using Selenium through Python
Solution 1:
As per your code trials on your Windows local machine non-headless Chrome works perfecto while on Linux server centos7 using headless Chrome you are redirected to the Access Denied page.
<html><head>\n<title>Access Denied</title>\n</head><body>\n<h1>Access Denied</h1>\n \nYou don\'t have permission to access "http://www.newark.com/" on this server.<p>\nReference #18.456cd417.1576243477.e007b9f\n\n\n</p></body></html>
Access Denied
As per the article How to bypass “Access Denied” pages when using Headless Chrome there is a little difference between Chrome when run in headless and when run in headed mode. The core network stack being the same and there being no differences in how the browser transmits requests at the packet level, points us to the content of the request only. On inspecting the requests made from headless and headed Chrome, it was observed headless Chrome is making itself known through it's User-Agent header. The header for headed Chrome was almost similar minus the Headless.
The headless Chrome User-Agent was:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/71.0.3578.98 Safari/537.36
Solution
So a precise solution would be to set the user-agent as of headed Chrome. The User-Agent for Chrome v79.x being:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36
You can modify your code as follows and execute:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument(f'user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36')
driver = webdriver.Chrome(executable_path="/usr/local/bin/chromedriver", chrome_options=options)
driver.get("https://www.google.com")
s = driver.page_source
print((s.encode("utf-8")))
driver.quit()
Executing on Windows 10 OS
Result of executing on windows-10 box:
-
Code Block:
from selenium import webdriver options = webdriver.ChromeOptions() options.add_argument("start-maximized") options.add_argument('--headless') options.add_argument('--no-sandbox') options.add_argument(f'user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36') options.add_experimental_option("excludeSwitches", ["enable-automation"]) options.add_experimental_option('useAutomationExtension', False) driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe') driver.get("https://www.google.com") s = driver.page_source print((s.encode("utf-8"))) driver.quit() -
Console Output:
[1214/041553.069:INFO:CONSOLE(0)] "A cookie associated with a resource at http://google.com/ was set with `SameSite=None` but without `Secure`. A future release of Chrome will only deliver cookies marked `SameSite=None` if they are also marked `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5633521622188032.", source: data:, (0) [1214/041553.165:INFO:CONSOLE(0)] "A cookie associated with a resource at http://google.com/ was set with `SameSite=None` but without `Secure`. A future release of Chrome will only deliver cookies marked `SameSite=None` if they are also marked `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5633521622188032.", source: https://www.google.com/ (0) [1214/041555.336:INFO:CONSOLE(0)] "A cookie associated with a cross-site resource at http://google.co.in/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.", source: https://www.google.com/ (0) b'<html itemscope="" itemtype="http://schema.org/WebPage" lang="en-IN"><head><meta charset="UTF-8"><meta content="origin" name="referrer"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><meta content="origin" name="referrer"><title>Google</title><script src="https://apis.google.com/_/scs/abc-static/_/js/k=gapi.gapi.en.7kWSr24wXFc.O/m=gapi_iframes,googleapis_client,plusone/rt=j/sv=1/d=1/ed=1/rs=AHpOoo-i9r7IbCTUQfJ0v-FPhRKRS8aihQ/cb=gapi.loaded_0" nonce="M5SQ7iANEaS7TdDvxjlnQA==" async=""></script><script nonce="M5SQ7iANEaS7TdDvxjlnQA==">(function(){window.google={kEI:\'0xr0Xe_oJNiV4-EP3NWg2As\',kEXPI:\'31\',authuser:0,kscs:\'c9c918f0_0xr0Xe_oJNiV4-EP3NWg2As\',kGL:\'IN\',kBL:\'9hVE\'};google.sn=\'webhp\';google.kHL=\'en-IN\';google.jsfs=\'Ffpdje\';})();(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI};google.getLEI=function(a){for(var b=null;a&&(!a.getAttribute||!(b=a.getAttribute("leid")));)a=a.parentNode;return b};google.https=function(){return"https:"==window.location.protocol};google.ml=function(){return null};google.time=function(){return(new Date).getTime()};google.log=function(a,b,e,c,g){if(a=google.logUrl(a,b,e,c,g)){b=new Image;var d=google.lc,f=google.li;d[f]=b;b.onerror=b.onload=b.onabort=function(){delete d[f]};google.vel&&google.vel.lu&&google.vel.lu(a);b.src=a;google.li=f+1}};google.logUrl=function(a,b,e,c,g){var d="",f=google.ls||"";e||-1!=b.search("&ei=")||(d="&ei="+google.getEI(c),-1==b.search("&lei=")&&(c=google.getLEI(c))&&(d+="&lei="+c));c="";!e&&google.cshid&&-1==b.search("&cshid=")&&"slh"!=a&&(c="&cshid="+google.cshid);a=e||"/"+(g||"gen_204")+"?atyp=i&ct="+a+"&cad="+b+d+f+"&zx="+google.time()+c;/^http:/i.test(a)&&google.https()&&(google.ml(Error("a"),!1,{src:a,glmm:1}),a="");return a};}).call(this);(function(){google.y={};google.x=function(a,b){if(a)var c=a.id;else{do c=Math.random();while(google.y[c])}google.y[c]=[a,b];return!1};google.lm=[];google.plm=function(a){google.lm.push.apply(google.lm,a)};google.lq=[];google.load=function(a,b,c){google.lq.push([[a],b,c])};google.loadAll=function(a,b){google.lq.push([a,b])};}).call(this);google.f={};(function(){document.documentElement.addEventListener("submit",function(b){var a;if(a=b.target){var c=a.getAttribute("data-submitfalse");a="1"==c||"q"==c&&!a.elements.q.value?!0:!1}else a=!1;a&&(b.preventDefault(),b.stopPropagation())},!0);}).call(this);(function(){google.hs={h:true,sie:false};})();(function(){google.c={gl:false,lhc:false};(function(){var e=window.performance;var g=function(a,b,c,d){a.addEventListener?a.removeEventListener(b,c,d||!1):a.attachEvent&&a.detachEvent("on"+b,c)},h=function(a,b,c,d){a.addEventListener?a.addEventListener(b,c,d||!1):a.attachEvent&&a.attachEvent("on"+b,c)};google.timers={};google.startTick=function(a){google.timers[a]={t:{start:google.time()},e:{},m:{}}};google.tick=function(a,b,c){google.timers[a]||google.startTick(a);c=void 0!==c?c:google.time();b instanceof Array||(b=[b]);for(var d=0,f;f=b[d++];)google.timers[a].t[f]=c};google.c.e=function(a,b,c){google.timers[a].e[b]=c};google.c.b=function(a){var b=google.timers.load.m;b[a]&&google.ml(Error("a"),!1,{m:a});b[a]=!0};google.c.u=function(a){var b=google.timers.load.m;if(b[a]){b[a]=!1;for(a in b)if(b[a])return;google.csiReport()}else google.ml(Error("b"),!1,{m:a})};google.rll=function(a,b,c){var d=function(f){c(f);g(a,"load",d);g(a,"error",d)};h(a,"load",d);b&&h(a,"error",d)};google.aft=function(a){a.setAttribute("data-iml",google.time())};google.startTick("load");var k=google.timers.load;a:{var l=k.t;if(e){var m=e.timing;if(m){var n=m.navigationStart,p=m.responseStart;if(p>n&&p<=l.start){l.start=p;k.wsrt=p-n;break a}}e.now&&(k.wsrt=Math.floor(e.now()))}}google.c.b("pr");google.c.b("xe");if(google.c.gl){var q=function(a){a&&google.aft(a.target)};h(document.documentElement,"load",q,!0);google.c.glu=function(){g(document.documentElement,"load",q,!0)}};}).call(this);})();(function(){var b=[function(){google.tick&&google.tick("load","dcl")}];google.dclc=function(a){b.length?b.push(a):a()};function c(){for(var a;a=b.shift();)a()}window.addEventListener?(document.addEventListener("DOMContentLoaded",c,!1),window.addEventListener("load",c,!1)):window.attachEvent&&window.attachEvent("onload",c);}).call(this);(function(){var b=[];google.jsc={xx:b,x:function(a){b.push(a)},mm:[],m:function(a){google.jsc.mm.length||(google.jsc.mm=a)}};}).call(this);(function(){var f=this||self,g=Date.now||function(){return+new Date};var y={};var aa=function(a,c){if(null===c)return!1;if("contains"in a&&1==c.nodeType)return a.contains(c);if("compareDocumentPosition"in a)return a==c||!!(a.compareDocumentPosition(c)&16);for(;c&&a!=c;)c=c.parentNode;return c==a};var ba=function(a,c){return function(d){d||(d=window.event);return c.call(a,d)}},A=function(a){a=a.target||a.srcElement;!a.getAttribute&&a.parentNode&&(a=a.parentNode);return a},B="undefined"!=typeof navigator&&/Macintosh/.test(navigator.userAgent),ca="undefined"!=typeof navigator&&!/Opera/.test(navigator.userAgent)&&/WebKit/.test(navigator.userAgent),da={A:1,INPUT:1,TEXTAREA:1,SELECT:1,BUTTON:1},ea=function(){this._mouseEventsPrevented=!0},F={A:13,BUTTON:0,CHECKBOX:32,COMBOBOX:13,FILE:0,GRIDCELL:13,LINK:13,LISTBOX:13,MENU:0,MENUBAR:0,MENUITEM:0,MENUITEMCHECKBOX:0,MENUITEMRADIO:0,OPTION:0,RADIO:32,RADIOGROUP:32,RESET:0,SUBMIT:0,SWITCH:32,TAB:0,TREE:13,TREEITEM:13},G={CHECKBOX:!0,FILE:!0,OPTION:!0,RADIO:!0},H={COLOR:!0,DATE:!0,DATETIME:!0,"DATETIME-LOCAL":!0,EMAIL:!0,MONTH:!0,NUMBER:!0,PASSWORD:!0,RANGE:!0,SEARCH:!0,TEL:!0,TEXT:!0,TEXTAREA:!0,TIME:!0,URL:!0,WEEK:!0},fa={A:!0,AREA:!0,BUTTON:!0,DIALOG:!0,IMG:!0,INPUT:!0,LINK:!0,MENU:!0,OPTGROUP:!0,OPTION:!0,PROGRESS:!0,SELECT:!0,TEXTAREA:!0};var I=function(){this.h=this.a=null},K=function(a,c){var d=J;d.a=a;d.h=c;return d};I.prototype.g=function(){var a=this.a;this.a&&this.a!=this.h?this.a=this.a.__owner||this.a.parentNode:this.a=null;return a};var L=function(){this.i=[];this.a=0;this.h=null;this.j=!1};L.prototype.g=function(){if(this.j)return J.g();if(this.a!=this.i.length){var a=this.i[this.a];this.a++;a!=this.h&&a&&a.__owner&&(this.j=!0,K(a.__owner,this.h));return a}return null};var J=new I,M=new L;var O=function(){this.o=[];this.a=[];this.g=[];this.j={};this.h=null;this.i=[];N(this,"_custom")},ha="undefined"!=typeof navigator&&/iPhone|iPad|iPod/.test(navigator.userAgent),P=String.prototype.trim?function(a){return a.trim()}:function(a){return a.replace(/^\\s+/,"").replace(/\\s+$/,"")},ia=/\\s*;\\s*/,ma=function(a,c){return function p(b,l){l=void 0===l?!0:l;var m=c;if("_custom"==m){m=b.detail;if(!m||!m._type)return;m=m._type}if("click"==m&&(B&&b.metaKey||!B&&b.ctrlKey||2==b.which||null==b.which&&\n4==b.button||b.shiftKey))m="clickmod";else{var k=b.which||b.keyCode;ca&&3==k&&(k=13);if(13!=k&&32!=k)k=!1;else{var e=A(b),n;(n="keydown"!=b.type||!!(!("getAttribute"in e)||(e.getAttribute("type")||e.tagName).toUpperCase()in H||"BUTTON"==e.tagName.toUpperCase()||e.type&&"FILE"==e.type.toUpperCase()||e.isContentEditable)||b.ctrlKey||b.shiftKey||b.altKey||b.metaKey||(e.getAttribute("type")||e.tagName).toUpperCase()in G&&32==k)||((n=e.tagName in da)||(n=e.getAttributeNode("tabindex"),n=null!=n&&n.specified),n=!(n&&!e.disabled));if(n)k=!1;else{n=(e.getAttribute("role")||e.type||e.tagName).toUpperCase();var q=!(n in F)&&13==k;e="INPUT"!=e.tagName.toUpperCase()||!!e.type;k=(0==F[n]%k||q)&&e}}k&&(m="clickkey")}e=b.srcElement||b.target;k=Q(m,b,e,"",null);b.path?(M.i=b.path,M.a=0,M.h=this,M.j=!1,n=M):n=K(e,this);for(;q=n.g();){var h=q;var r=void 0;var v=h;q=m;var t=v.__jsaction;if(!t){var z;t=null;"getAttribute"in v&&(t=v.getAttribute("jsaction"));if(z=t){t=y[z];if(!t){t={};for(var C=z.split(ia),ja=C?C.length:0,D=0;D<ja;D++){var x=C[D];if(x){var E=x.indexOf(":"),R=-1!=E,ka=R?P(x.substr(0,E)):"click";x=R?P(x.substr(E+1)):x;t[ka]=x}}y[z]=t}v.__jsaction=t}else t=la,v.__jsaction=t}v=t;"maybe_click"==q&&v.click?(r=q,q="click"):"clickkey"==q?q="click":"click"!=q||v.click||(q="clickonly");r={m:r?r:q,action:v[q]||"",event:null,s:!1};k=Q(r.m,r.event||b,e,r.action||"",h,k.timeStamp);if(r.s||r.action)break}k&&"touchend"==k.eventType&&(k.event._preventMouseEvents=ea);if(r&&r.action){if(e="clickkey"==m)e=A(b),e=(e.type||\ne.tagName).toUpperCase(),(e=32==(b.which||b.keyCode)&&"CHECKBOX"!=e)||(e=A(b),n=e.tagName.toUpperCase(),r=(e.getAttribute("role")||"").toUpperCase(),e="BUTTON"===n||"BUTTON"===r?!0:!(e.tagName.toUpperCase()in fa)||"A"===n||"SELECT"===n||(e.getAttribute("type")||e.tagName).toUpperCase()in G||(e.getAttribute("type")||e.tagName).toUpperCase()in H?!1:!0);e&&(b.preventDefault?b.preventDefault():b.returnValue=!1);if("mouseenter"==m||"mouseleave"==m)if(e=b.relatedTarget,!("mouseover"==b.type&&"mouseenter"==\nm||"mouseout"==b.type&&"mouseleave"==m)||e&&(e===h||aa(h,e)))k.action="",k.actionElement=null;else{m={};for(var u in b)"function"!==typeof b[u]&&"srcElement"!==u&&"target"!==u&&(m[u]=b[u]);m.type="mouseover"==b.type?"mouseenter":"mouseleave";m.target=m.srcElement=h;m.bubbles=!1;k.event=m;k.targetElement=h}}else k.action="",k.actionElement=null;h=k;a.h&&!h.event.a11ysgd&&(u=Q(h.eventType,h.event,h.targetElement,h.action,h.actionElement,h.timeStamp),"clickonly"==u.eventType&&(u.eventType="click"),a.h(u,!0));if(h.actionElement){if(a.h){!h.actionElement||"A"!=h.actionElement.tagName||"click"!=h.eventType&&"clickmod"!=h.eventType||(b.preventDefault?b.preventDefault():b.returnValue=!1);var w=a.h(h);if(w&&l){p.call(this,w,!1);return}}else{if((u=f.document)&&!u.createEvent&&u.createEventObject)try{w=u.createEventObject(b)}catch(pa){w=b}else w=b;h.event=w;a.i.push(h)}if("touchend"==h.event.type&&h.event._mouseEventsPrevented){w=h.event;for(var qa in w);g()}}}},Q=function(a,c,d,b,l,p){return{eventType:a,event:c,targetElement:d,action:b,actionElement:l,timeStamp:p||g()}},la={},na=function(a,c){return function(d){var b=a,l=c,p=!1;"mouseenter"==b?b="mouseover":"mouseleave"==b&&(b="mouseout");if(d.addEventListener){if("focus"==b||"blur"==b||"error"==b||"load"==b)p=!0;d.addEventListener(b,l,p)}else d.attachEvent&&("focus"==b?b="focusin":"blur"==b&&(b="focusout"),l=ba(d,l),d.attachEvent("on"+b,l));return{m:b,l:l,capture:p}}},N=function(a,c){if(!a.j.hasOwnProperty(c)){var d=ma(a,c),b=na(c,d);a.j[c]=d;a.o.push(b);for(d=0;d<a.a.length;++d){var l=a.a[d];l.g.push(b.call(null,l.a))}"click"==c&&N(a,"keydown")}};O.prototype.l=function(a){return this.j[a]};var V=function(a,c){var d=new oa(c),b;a:{for(b=0;b<a.a.length;b++)if(S(a.a[b],c)){b=!0;break a}b=!1}if(b)return a.g.push(d),d;T(a,d);a.a.push(d);U(a);return d},U=function(a){for(var c=a.g.concat(a.a),d=[],b=[],l=0;l<a.a.length;++l){var p=a.a[l];W(p,c)?(d.push(p),X(p)):b.push(p)}for(l=0;l<a.g.length;++l)p=a.g[l],W(p,c)?d.push(p):(b.push(p),T(a,p));a.a=b;a.g=d},T=function(a,c){var d=c.a;ha&&(d.style.cursor="pointer");for(d=0;d<a.o.length;++d)c.g.push(a.o[d].call(null,c.a))},Y=function(a,c){a.h=c;a.i&&\n(0<a.i.length&&c(a.i),a.i=null)},oa=function(a){this.a=a;this.g=[]},S=function(a,c){for(var d=a.a,b=c;d!=b&&b.parentNode;)b=b.parentNode;return d==b},W=function(a,c){for(var d=0;d<c.length;++d)if(c[d].a!=a.a&&S(c[d],a.a))return!0;return!1},X=function(a){for(var c=0;c<a.g.length;++c){var d=a.a,b=a.g[c];d.removeEventListener?d.removeEventListener(b.m,b.l,b.capture):d.detachEvent&&d.detachEvent("on"+b.m,b.l)}a.g=[]};var Z=new O;V(Z,window.document.documentElement);N(Z,"click");N(Z,"focus");N(Z,"focusin");N(Z,"blur");N(Z,"focusout");N(Z,"error");N(Z,"load");N(Z,"change");N(Z,"dblclick");N(Z,"input");N(Z,"keyup");N(Z,"keydown");N(Z,"keypress");N(Z,"mousedown");N(Z,"mouseenter");N(Z,"mouseleave");N(Z,"mouseout");N(Z,"mouseover");N(Z,"mouseup");N(Z,"paste");N(Z,"touchstart");N(Z,"touchend");N(Z,"touchcancel");N(Z,"speech");(function(a){google.jsad=function(c){Y(a,c)};google.jsaac=function(c){return V(a,c)};google.jsarc=function(c){X(c);for(var d=!1,b=0;b<a.a.length;++b)if(a.a[b]===c){a.a.splice(b,1);d=!0;break}if(!d)for(d=0;d<a.g.length;++d)if(a.g[d]===c){a.g.splice(d,1);break}U(a)}})(Z);window.gws_wizbind=function(a){return{trigger:function(c){var d=a.l(c.type);d||(N(a,c.type),d=a.l(c.type));var b=c.target||c.srcElement;d&&d.call(b.ownerDocument.documentElement,c)},bind:function(c){Y(a,c)}}}(Z);}).call(this);(function(){window.jsarwt=function(){return!1};}).call(this);var a=window.location,b=a.href.indexOf("#");if(0<=b){var c=a.href.substring(b+1);/(^|&)q=/.test(c)&&-1==c.indexOf("#")&&a.replace("/search?"+c.replace(/(^|&)fp=[^&]*/g,"")+"&cad=h")};var k="function"==typeof Object.defineProperties?Object.defineProperty:function(a,e,c){a!=Array.prototype&&a!=Object.prototype&&(a[e]=c.value)},l="undefined"!=typeof window&&window===this?this:"undefined"!=typeof global&&null!=global?global:this,m=function(a,e){if(e){for(var c=l,b=a.split("."),f=0;f<b.length-1;f++){var d=b[f];d in c||(c[d]={});c=c[d]}b=b[b.length-1];f=c[b];d=e(f);d!=f&&null!=d&&k(c,b,{configurable:!0,writable:!0,value:d})}};m("String.prototype.startsWith",function(a){return a?a:function(e,c){if(null==this)throw new TypeError("The \'this\' value for String.prototype.startsWith must not be null or undefined");if(e instanceof RegExp)throw new TypeError("First argument to String.prototype.startsWith must not be a regular expression");var b=this+"";e+="";for(var f=b.length,d=e.length,h=Math.max(0,Math.min(c|0,b.length)),g=0;g<d&&h<f;)if(b[h++]!=e[g++])return!1;return g>=d}});google.arwt=function(a){a.href=document.getElementById(a.id.substring(a.id.startsWith("vcs")?3:1)).href;return!0};(function(){function e(a){if(!a||/[?&]dsh=1(&|$)/.test(a))return null;if(/[?&]ae=1(&|$)/.test(a)){var c=/[?&]adurl=([^&]+)/.exec(a);if(!c)return null;var . . . <textarea class="csi" name="csi" style="display:none"></textarea> . . . </script><script src="/xjs/_/js/k=xjs.s.en_GB.67AovsT7uRo.O/ck=xjs.s.8N3EJVB8puY.L.W.O/am=AAAAgEUAu-6AIP9vBQAAgB0DAAABbsEGC4QhocJYnYAIEA/d=1/exm=Fkg7bd,HcFEGb,IvlUe,MC8mtf,OF7gzc,RMhBfe,T4BAC,TJw5qb,TbaHGc,Y33vzc,cdos,csi,d,hsm,iDPoPb,jsa,mvYTse,tg8oTe,uz938c,vWNDde,ws9Tlc,yQ43ff/ed=1/dg=2/br=1/ct=zgms/rs=ACT90oGiS0w5o50czzw_k4qZG0GKuVBevw/m=WgDvvc,aa,abd,async,dvl,fEVMic,foot,lu,m,mUpTid,mu,sb_wiz,sf,sonic,spch,xz7cCd?xjs=s1" async="" gapi_processed="true"></script></body></html>'
tl; dr
You can find a relevant discussion in:
- Way to change Google Chrome user agent in Selenium?
Solution 2:
Some websites are denying access to headless browsers. On your local machine the browser does not run headless, on the linux machine it does. You can circumvent that by setting a custom user agent:
options.add_argument("user-agent=whatever you want")
If you set that to the user agent you are using on your windows machine, that should do the trick.