Unit testing a python app that uses the requests library

I am writing an application that performs REST operations using Kenneth Reitz's requests library and I'm struggling to find a nice way to unit test these applications, because requests provides its methods via module-level methods.

What I want is the ability to synthesize the conversation between the two sides; provide a series of request assertions and responses.

It is in fact a little strange that the library has a blank page about end-user unit testing, while targeting user-friendliness and ease of use. There's however an easy-to-use library by Dropbox, unsurprisingly called responses. Here is its intro post. It says they've failed to employ httpretty, while stating no reason of the fail, and written a library with similar API.

import unittest

import requests
import responses


class TestCase(unittest.TestCase):

  @responses.activate  
  def testExample(self):
    responses.add(**{
      'method'         : responses.GET,
      'url'            : 'http://example.com/api/123',
      'body'           : '{"error": "reason"}',
      'status'         : 404,
      'content_type'   : 'application/json',
      'adding_headers' : {'X-Foo': 'Bar'}
    })

    response = requests.get('http://example.com/api/123')

    self.assertEqual({'error': 'reason'}, response.json())
    self.assertEqual(404, response.status_code)

If you use specifically requests try httmock. It's wonderfully simple and elegant:

from httmock import urlmatch, HTTMock
import requests

# define matcher:
@urlmatch(netloc=r'(.*\.)?google\.com$')
def google_mock(url, request):
    return 'Feeling lucky, punk?'

# open context to patch
with HTTMock(google_mock):
    # call requests
    r = requests.get('http://google.com/')
print r.content  # 'Feeling lucky, punk?'

If you want something more generic (e.g. to mock any library making http calls) go for httpretty.

Almost as elegant:

import requests
import httpretty

@httpretty.activate
def test_one():
    # define your patch:
    httpretty.register_uri(httpretty.GET, "http://yipit.com/",
                        body="Find the best daily deals")
    # use!
    response = requests.get('http://yipit.com')
    assert response.text == "Find the best daily deals"

HTTPretty is far more feature-rich - it offers also mocking status code, streaming responses, rotating responses, dynamic responses (with a callback).

You could use a mocking library such as Mocker to intercept the calls to the requests library and return specified results.

As a very simple example, consider this class which uses the requests library:

class MyReq(object):
    def doSomething(self):
        r = requests.get('https://api.github.com', auth=('user', 'pass'))
        return r.headers['content-type']

Here's a unit test that intercepts the call to requests.get and returns a specified result for testing:

import unittest
import requests
import myreq

from mocker import Mocker, MockerTestCase

class MyReqTests(MockerTestCase):
    def testSomething(self):
        # Create a mock result for the requests.get call
        result = self.mocker.mock()
        result.headers
        self.mocker.result({'content-type': 'mytest/pass'})

        # Use mocker to intercept the call to requests.get
        myget = self.mocker.replace("requests.get")
        myget('https://api.github.com', auth=('user', 'pass'))
        self.mocker.result(result)

        self.mocker.replay()

        # Now execute my code
        r = myreq.MyReq()
        v = r.doSomething()

        # and verify the results
        self.assertEqual(v, 'mytest/pass')
        self.mocker.verify()

if __name__ == '__main__':
    unittest.main()

When I run this unit test I get the following result:

.
----------------------------------------------------------------------
Ran 1 test in 0.004s

OK