How to set proxy authentication (user & password) using Python + Selenium

I am using Firefox WebDriver in Python 2.7 with Selenium. My python program starts Firefox browser and visits different websites when I run the program. But, I need to set the proxy with authentication, so that when program visits any website, it will visit through the proxy server.

There are some similar questions on SO. But, there is no specific solution for Selenium Firefox WebDriver of Python.

• Python Selenium Webdriver - Proxy Authentication
• Running selenium behind a proxy server

Solution 1:

Dup of: How to set proxy AUTHENTICATION username:password using Python/Selenium

Selenium-wire: https://github.com/wkeeling/selenium-wire

Install selenium-wire

pip install selenium-wire

Import it

from seleniumwire import webdriver

Auth to proxy

options = {
'proxy': {
    'http': 'http://username:password@host:port',
    'https': 'https://username:password@host:port',
    'no_proxy': 'localhost,127.0.0.1,dev_server:8080'
    }
}
driver = webdriver.Firefox(seleniumwire_options=options)

Warning
Take a look to the selenium-wire cache folder. I had a problem because it take all my disk space. You have to remove it sometimes in your script when you want.

Solution 2:

In addition to running Firefox with a profile which has the credentials saved. You can do it loading an extension that writes in the loginTextbox and password1Textbox of chrome://global/content/commonDialog.xul (the alert window).

There are already some extensions that will do the job. For instance: Close Proxy Authentication

https://addons.mozilla.org/firefox/downloads/latest/close-proxy-authentication/addon-427702-latest.xpi

from selenium import webdriver
from base64 import b64encode

proxy = {'host': HOST, 'port': PORT, 'usr': USER, 'pwd': PASSWD}

fp = webdriver.FirefoxProfile()

fp.add_extension('closeproxy.xpi')
fp.set_preference('network.proxy.type', 1)
fp.set_preference('network.proxy.http', proxy['host'])
fp.set_preference('network.proxy.http_port', int(proxy['port']))
# ... ssl, socks, ftp ...
fp.set_preference('network.proxy.no_proxies_on', 'localhost, 127.0.0.1')

credentials = '{usr}:{pwd}'.format(**proxy)
credentials = b64encode(credentials.encode('ascii')).decode('utf-8')
fp.set_preference('extensions.closeproxyauth.authtoken', credentials)

driver = webdriver.Firefox(fp)

Solution 3:

You can write own firefox extension for proxy, and launch from selenium. You need write 2 files and pack it.

background.js

var proxy_host = "YOUR_PROXY_HOST";
var proxy_port = YOUR_PROXY_PORT;

var config = {
    mode: "fixed_servers",
    rules: {
      singleProxy: {
        scheme: "http",
        host: proxy_host,
        port: proxy_port
      },
      bypassList: []
    }
 };


function proxyRequest(request_data) {
    return {
        type: "http",
        host: proxy_host, 
        port: proxy_port
    };
}

browser.proxy.settings.set({value: config, scope: "regular"}, function() {;});

function callbackFn(details) {
return {
    authCredentials: {
        username: "YOUR_USERNAME",
        password: "YOUR_PASSWORD"
    }
};
}

browser.webRequest.onAuthRequired.addListener(
        callbackFn,
        {urls: ["<all_urls>"]},
        ['blocking']
);

browser.proxy.onRequest.addListener(proxyRequest, {urls: ["<all_urls>"]});

manifest.json

{
  "name": "My Firefox Proxy",
  "version": "1.0.0b",
  "manifest_version": 2,
  "permissions": [
    "browsingData",
    "proxy",
    "storage",
    "tabs",
    "webRequest",
    "webRequestBlocking",
    "downloads",
    "notifications",
    "<all_urls>"
  ],
  "background": {
    "scripts": ["background.js"]
  },
  "browser_specific_settings": {
    "gecko": {
      "id": "[email protected]"
    }
  }
}

Next you need packed this files to zip archive in DEFLATED mode with .xpi at end like my_proxy_extension.xpi.

You have two choices:

1. Sign your extension Firefox extension .xpi file structure: description, contents, creation, and installation

OR

2. Run unsigned. For this step:

   1. Open firefox flags at about:config and set options xpinstall.signatures.required to false

      OR

   2. Update firefox profile in:

      Windows: C:\Program Files\Mozilla Firefox\defaults\pref\channel-prefs.js

      Linux: /etc/firefox/syspref.js

   Add next line to end of file:

    pref("xpinstall.signatures.required",false);
   

After this steps run selenium and install this extension:

from selenium import webdriver

driver = webdriver.Firefox()
driver.install_addon("path/to/my_proxy_extension.xpi")

driver.get("https://yoursite.com")