How could I use requests in asyncio?
Solution 1:
To use requests (or any other blocking libraries) with asyncio, you can use BaseEventLoop.run_in_executor to run a function in another thread and yield from it to get the result. For example:
import asyncio
import requests
@asyncio.coroutine
def main():
loop = asyncio.get_event_loop()
future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
response1 = yield from future1
response2 = yield from future2
print(response1.text)
print(response2.text)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
This will get both responses in parallel.
With python 3.5 you can use the new await
/async
syntax:
import asyncio
import requests
async def main():
loop = asyncio.get_event_loop()
future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
response1 = await future1
response2 = await future2
print(response1.text)
print(response2.text)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
See PEP0492 for more.
Solution 2:
aiohttp can be used with HTTP proxy already:
import asyncio
import aiohttp
@asyncio.coroutine
def do_request():
proxy_url = 'http://localhost:8118' # your proxy address
response = yield from aiohttp.request(
'GET', 'http://google.com',
proxy=proxy_url,
)
return response
loop = asyncio.get_event_loop()
loop.run_until_complete(do_request())
Solution 3:
The answers above are still using the old Python 3.4 style coroutines. Here is what you would write if you got Python 3.5+.
aiohttp
supports http proxy now
import aiohttp
import asyncio
async def fetch(session, url):
async with session.get(url) as response:
return await response.text()
async def main():
urls = [
'http://python.org',
'https://google.com',
'http://yifei.me'
]
tasks = []
async with aiohttp.ClientSession() as session:
for url in urls:
tasks.append(fetch(session, url))
htmls = await asyncio.gather(*tasks)
for html in htmls:
print(html[:100])
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
There is also the httpx
library, which is a drop in replacement for requests with async/await
support.
Solution 4:
Requests does not currently support asyncio
and there are no plans to provide such support. It's likely that you could implement a custom "Transport Adapter" (as discussed here) that knows how to use asyncio
.
If I find myself with some time it's something I might actually look into, but I can't promise anything.