Aiohttp tutorial

Aiohttp tutorial DEFAULT

Aiohttp tutorial¶

This tutorial shows how to build an REST API application following the dependency injection principle.

Start from the scratch or jump to the section:

You can find complete project on the Github.

What are we going to build?¶

https://media.giphy.com/media/apvx5lPCPsjN6/source.gif

We will build a REST API application that searches for funny GIFs on the Giphy. Let’s call it Giphy Navigator.

How does Giphy Navigator work?

  • Client sends a request specifying the search query and the number of results.

  • Giphy Navigator returns a response in json format.

  • The response contains:
    • the search query

    • the limit number

    • the list of gif urls

Example response:

{"query":"Dependency Injector","limit":10,"gifs":[{"url":"https://giphy.com/gifs/boxes-dependent-swbf2-6Eo7KzABxgJMY"},{"url":"https://giphy.com/gifs/depends-J56qCcOhk6hKE"},{"url":"https://giphy.com/gifs/web-series-ccstudios-bro-dependent-1lhU8KAVwmVVu"},{"url":"https://giphy.com/gifs/TheBoysTV-friends-friend-weneedeachother-XxR9qcIwcf5Jq404Sx"},{"url":"https://giphy.com/gifs/netflix-a-series-of-unfortunate-events-asoue-9rgeQXbwoK53pcxn7f"},{"url":"https://giphy.com/gifs/black-and-white-sad-skins-Hs4YzLs2zJuLu"},{"url":"https://giphy.com/gifs/always-there-for-you-i-am-here-PlayjhCco9jHBYrd9w"},{"url":"https://giphy.com/gifs/stream-famous-dollar-YT2dvOByEwXCdoYiA1"},{"url":"https://giphy.com/gifs/i-love-you-there-for-am-1BhGzgpZXYWwWMAGB1"},{"url":"https://giphy.com/gifs/life-like-twerk-9hlnWxjHqmH28"}]}

The task is naive and that’s exactly what we need for the tutorial.

Prepare the environment¶

Let’s create the environment for the project.

First we need to create a project folder:

mkdir giphynav-aiohttp-tutorial cd giphynav-aiohttp-tutorial

Now let’s create and activate virtual environment:

python3 -m venv venv . venv/bin/activate

Environment is ready and now we’re going to create the layout of the project.

Project layout¶

Create next structure in the current directory. All files should be empty. That’s ok for now.

Initial project layout:

./ ├── giphynavigator/ │ ├── __init__.py │ ├── application.py │ ├── containers.py │ └── handlers.py ├── venv/ └── requirements.txt

Install the requirements¶

Now it’s time to install the project requirements. We will use next packages:

  • - the dependency injection framework

  • - the web framework

  • - the helper library that will provide a development server with live reloading

  • - the YAML files parsing library, used for the reading of the configuration files

  • - the helper library for the testing of the application

  • - the helper library for measuring the test coverage

Put next lines into the file:

dependency-injector aiohttp aiohttp-devtools pyyaml pytest-aiohttp pytest-cov

and run next in the terminal:

pip install -r requirements.txt

Let’s also install the . It is a user-friendly command-line HTTP client for the API era. We will use it for the manual testing.

Run the command in the terminal:

The requirements are setup. Now we will build a minimal application.

Minimal application¶

In this section we will build a minimal application. It will have an endpoint that will answer our requests in json format. There will be no payload for now.

Edit :

"""Handlers module."""fromaiohttpimportwebasyncdefindex(request:web.Request)->web.Response:query=request.query.get('query','Dependency Injector')limit=int(request.query.get('limit',10))gifs=[]returnweb.json_response({'query':query,'limit':limit,'gifs':gifs,},)

Now let’s create a container. Container will keep all of the application components and their dependencies.

Edit :

"""Containers module."""fromdependency_injectorimportcontainersclassContainer(containers.DeclarativeContainer):...

Container is empty for now. We will add the providers in the following sections.

Finally we need to create application factory. It will create and configure container and . It is traditionally called . We will assign handler to handle user requests to the root of our web application.

Put next into the :

"""Application module."""fromaiohttpimportwebfrom.containersimportContainerfrom.importhandlersdefcreate_app()->web.Application:container=Container()app=web.Application()app.container=containerapp.add_routes([web.get('/',handlers.index),])returnapp

Now we’re ready to run our application

Do next in the terminal:

adev runserver giphynavigator/application.py --livereload

The output should be something like:

[18:52:59] Starting aux server at http://localhost:8001 ◆ [18:52:59] Starting dev server at http://localhost:8000 ●

Let’s check that it works. Open another terminal session and use :

http http://127.0.0.1:8000/

You should see:

HTTP/1.1 200 OK Content-Length: 844 Content-Type: application/json; charset=utf-8 Date: Wed, 29 Jul 2020 21:01:50 GMT Server: Python/3.8 aiohttp/3.6.2 { "gifs": [], "limit": 10, "query": "Dependency Injector" }

Minimal application is ready. Let’s connect our application with the Giphy API.

Giphy API client¶

In this section we will integrate our application with the Giphy API.

We will create our own API client using client.

Create module in the package:

./ ├── giphynavigator/ │ ├── __init__.py │ ├── application.py │ ├── containers.py │ ├── giphy.py │ └── handlers.py ├── venv/ └── requirements.txt

and put next into it:

"""Giphy client module."""fromaiohttpimportClientSession,ClientTimeoutclassGiphyClient:API_URL='https://api.giphy.com/v1'def__init__(self,api_key,timeout):self._api_key=api_keyself._timeout=ClientTimeout(timeout)asyncdefsearch(self,query,limit):"""Make search API call and return result."""url=f'{self.API_URL}/gifs/search'params={'q':query,'api_key':self._api_key,'limit':limit,}asyncwithClientSession(timeout=self._timeout)assession:asyncwithsession.get(url,params=params)asresponse:ifresponse.status!=200:response.raise_for_status()returnawaitresponse.json()

Now we need to add into the container. The has two dependencies that have to be injected: the API key and the request timeout. We will need to use two more providers from the module:

  • provider that will create the client.

  • provider that will provide the API key and the request timeout.

Edit :

"""Containers module."""fromdependency_injectorimportcontainers,providersfrom.importgiphyclassContainer(containers.DeclarativeContainer):config=providers.Configuration()giphy_client=providers.Factory(giphy.GiphyClient,api_key=config.giphy.api_key,timeout=config.giphy.request_timeout,)

Note

We have used the configuration value before it was defined. That’s the principle how the provider works.

Use first, define later.

Now let’s add the configuration file.

We will use YAML.

Create an empty file in the root root of the project:

./ ├── giphynavigator/ │ ├── __init__.py │ ├── application.py │ ├── containers.py │ ├── giphy.py │ └── handlers.py ├── venv/ ├── config.yml └── requirements.txt

and put next into it:

giphy:request_timeout:10

We will use an environment variable to provide the API key.

Now we need to edit to make two things when application starts:

  • Load the configuration file the .

  • Load the API key from the environment variable.

Edit :

"""Application module."""fromaiohttpimportwebfrom.containersimportContainerfrom.importhandlersdefcreate_app()->web.Application:container=Container()container.config.from_yaml('config.yml')container.config.giphy.api_key.from_env('GIPHY_API_KEY')app=web.Application()app.container=containerapp.add_routes([web.get('/',handlers.index),])returnapp

Now we need to create an API key and set it to the environment variable.

As for now, don’t worry, just take this one:

exportGIPHY_API_KEY=wBJ2wZG7SRqfrU9nPgPiWvORmloDyuL0

Note

To create your own Giphy API key follow this guide.

The Giphy API client and the configuration setup is done. Let’s proceed to the search service.

Search service¶

Now it’s time to add the . It will:

  • Perform the search.

  • Format result data.

will use .

Create module in the package:

./ ├── giphynavigator/ │ ├── __init__.py │ ├── application.py │ ├── containers.py │ ├── giphy.py │ ├── handlers.py │ └── services.py ├── venv/ ├── config.yml └── requirements.txt

and put next into it:

"""Services module."""from.giphyimportGiphyClientclassSearchService:def__init__(self,giphy_client:GiphyClient):self._giphy_client=giphy_clientasyncdefsearch(self,query,limit):"""Search for gifs and return formatted data."""ifnotquery:return[]result=awaitself._giphy_client.search(query,limit)return[{'url':gif['url']}forgifinresult['data']]

The has a dependency on the . This dependency will be injected when we add to the container.

Edit :

"""Containers module."""fromdependency_injectorimportcontainers,providersfrom.importgiphy,servicesclassContainer(containers.DeclarativeContainer):config=providers.Configuration()giphy_client=providers.Factory(giphy.GiphyClient,api_key=config.giphy.api_key,timeout=config.giphy.request_timeout,)search_service=providers.Factory(services.SearchService,giphy_client=giphy_client,)

The search service is ready. In next section we’re going to put it to work.

Make the search work¶

Now we are ready to put the search into work. Let’s inject into the handler. We will use Wiring feature.

Edit :

"""Handlers module."""fromaiohttpimportwebfromdependency_injector.wiringimportinject,Provid[email protected]injectasyncdefindex(request:web.Request,search_service:SearchService=Provide[Container.search_service],)->web.Response:query=request.query.get('query','Dependency Injector')limit=int(request.query.get('limit',10))gifs=awaitsearch_service.search(query,limit)returnweb.json_response({'query':query,'limit':limit,'gifs':gifs,},)

To make the injection work we need to wire the container instance with the module. This needs to be done once. After it’s done we can use markers to specify as many injections as needed for any handler.

Edit :

"""Application module."""fromaiohttpimportwebfrom.containersimportContainerfrom.importhandlersdefcreate_app()->web.Application:container=Container()container.config.from_yaml('config.yml')container.config.giphy.api_key.from_env('GIPHY_API_KEY')container.wire(modules=[handlers])app=web.Application()app.container=containerapp.add_routes([web.get('/',handlers.index),])returnapp

Make sure the app is running or use:

adev runserver giphynavigator/application.py --livereload

and make a request to the API in the terminal:

http http://localhost:8000/ query=="wow,it works"limit==5

You should see:

HTTP/1.1 200 OK Content-Length: 492 Content-Type: application/json; charset=utf-8 Date: Fri, 09 Oct 2020 01:35:48 GMT Server: Python/3.8 aiohttp/3.6.2 { "gifs": [ { "url": "https://giphy.com/gifs/dollyparton-3xIVVMnZfG3KQ9v4Ye" }, { "url": "https://giphy.com/gifs/tennistv-unbelievable-disbelief-cant-believe-UWWJnhHHbpGvZOapEh" }, { "url": "https://giphy.com/gifs/discoverychannel-nugget-gold-rush-rick-ness-KGGPIlnC4hr4u2s3pY" }, { "url": "https://giphy.com/gifs/soulpancake-wow-work-xUe4HVXTPi0wQ2OAJC" }, { "url": "https://giphy.com/gifs/readingrainbow-teamwork-levar-burton-reading-rainbow-3o7qE1EaTWLQGDSabK" } ], "limit": 5, "query": "wow,it works" }
https://media.giphy.com/media/3oxHQCI8tKXoeW4IBq/source.gif

The search works!

Make some refactoring¶

Our handler has two hardcoded config values:

  • Default search query

  • Default results limit

Let’s make some refactoring. We will move these values to the config.

Edit :

"""Handlers module."""fromaiohttpimportwebfromdependency_injector.wiringimportinject,Provid[email protected]injectasyncdefindex(request:web.Request,search_service:SearchService=Provide[Container.search_service],default_query:str=Provide[Container.config.default.query],default_limit:int=Provide[Container.config.default.limit.as_int()],)->web.Response:query=request.query.get('query',default_query)limit=int(request.query.get('limit',default_limit))gifs=awaitsearch_service.search(query,limit)returnweb.json_response({'query':query,'limit':limit,'gifs':gifs,},)

Let’s update the config.

Edit :

giphy:request_timeout:10default:query:"DependencyInjector"limit:10

The refactoring is done. We’ve made it cleaner - hardcoded values are now moved to the config.

Tests¶

In this section we will add some tests.

Create module in the package:

./ ├── giphynavigator/ │ ├── __init__.py │ ├── application.py │ ├── containers.py │ ├── giphy.py │ ├── handlers.py │ ├── services.py │ └── tests.py ├── venv/ ├── config.yml └── requirements.txt

and put next into it:

"""Tests module."""fromunittestimportmockimportpytestfromgiphynavigator.appli[email protected]pytest.fixturedefapp():app=create_app()yieldappapp.container.unwire()@pytest.fixturedefclient(app,aiohttp_client,loop):returnloop.run_until_complete(aiohttp_client(app))asyncdeftest_index(client,app):giphy_client_mock=mock.AsyncMock(spec=GiphyClient)giphy_client_mock.search.return_value={'data':[{'url':'https://giphy.com/gif1.gif'},{'url':'https://giphy.com/gif2.gif'},],}withapp.container.giphy_client.override(giphy_client_mock):response=awaitclient.get('/',params={'query':'test','limit':10,},)assertresponse.status==200data=awaitresponse.json()assertdata=={'query':'test','limit':10,'gifs':[{'url':'https://giphy.com/gif1.gif'},{'url':'https://giphy.com/gif2.gif'},],}asyncdeftest_index_no_data(client,app):giphy_client_mock=mock.AsyncMock(spec=GiphyClient)giphy_client_mock.search.return_value={'data':[],}withapp.container.giphy_client.override(giphy_client_mock):response=awaitclient.get('/')assertresponse.status==200data=awaitresponse.json()assertdata['gifs']==[]asyncdeftest_index_default_params(client,app):giphy_client_mock=mock.AsyncMock(spec=GiphyClient)giphy_client_mock.search.return_value={'data':[],}withapp.container.giphy_client.override(giphy_client_mock):response=awaitclient.get('/')assertresponse.status==200data=awaitresponse.json()assertdata['query']==app.container.config.default.query()assertdata['limit']==app.container.config.default.limit()

Now let’s run it and check the coverage:

py.test giphynavigator/tests.py --cov=giphynavigator

You should see:

platformdarwin--Python3.8.3,pytest-5.4.3,py-1.9.0,pluggy-0.13.1plugins:cov-2.10.0,aiohttp-0.3.0,asyncio-0.14.0collected3itemsgiphynavigator/tests.py...[100%]----------coverage:platformdarwin,python3.8.3-final-0-----------NameStmtsMissCover---------------------------------------------------giphynavigator/__init__.py00100%giphynavigator/application.py120100%giphynavigator/containers.py60100%giphynavigator/giphy.py14936%giphynavigator/handlers.py100100%giphynavigator/services.py9189%giphynavigator/tests.py370100%---------------------------------------------------TOTAL881089%

Note

Take a look at the highlights in the .

It emphasizes the overriding of the . The real API call are mocked.

Conclusion¶

In this tutorial we’ve built an REST API application following the dependency injection principle. We’ve used the as a dependency injection framework.

Containers and Providers helped to specify how to assemble search service and giphy client.

Configuration provider helped to deal with reading YAML file and environment variable.

We used Wiring feature to inject the dependencies into the handler. Provider overriding feature helped in testing.

We kept all the dependencies injected explicitly. This will help when you need to add or change something in future.

You can find complete project on the Github.

What’s next?

Sours: https://python-dependency-injector.ets-labs.org/tutorials/aiohttp.html

Asynchronous HTTP Requests in Python with aiohttp and asyncio

Asynchronous code has increasingly become a mainstay of Python development. With asyncio becoming part of the standard library and many third party packages providing features compatible with it, this paradigm is not going away anytime soon.

Let's walk through how to use the aiohttp library to take advantage of this for making asynchronous HTTP requests, which is one of the most common use cases for non-blocking code.

What is non-blocking code?

You may hear terms like "asynchronous", "non-blocking" or "concurrent" and be a little confused as to what they all mean. According to this much more detailed tutorial, two of the primary properties are:

  • Asynchronous routines are able to “pause” while waiting on their ultimate result to let other routines run in the meantime.
  • Asynchronous code, through the mechanism above, facilitates concurrent execution. To put it differently, asynchronous code gives the look and feel of concurrency.

So asynchronous code is code that can hang while waiting for a result, in order to let other code run in the meantime. It doesn't "block" other code from running so we can call it "non-blocking" code.

The asyncio library provides a variety of tools for Python developers to do this, and aiohttp provides an even more specific functionality for HTTP requests. HTTP requests are a classic example of something that is well-suited to asynchronicity because they involve waiting for a response from a server, during which time it would be convenient and efficient to have other code running.

Setting up

Make sure to have your Python environment setup before we get started. Follow this guide up through the virtualenv section if you need some help. Getting everything working correctly, especially with respect to virtual environments is important for isolating your dependencies if you have multiple projects running on the same machine. You will need at least Python 3.7 or higher in order to run the code in this post.

Now that your environment is set up, you’re going to need to install some third party libraries. We’re going to use aiohttp for making asynchronous requests, and the requests library for making regular synchronous HTTP requests in order to compare the two later on. Install both of these with the following command after activating your virtual environment:

With this you should be ready to move on and write some code.

Making an HTTP Request with aiohttp

Let's start off by making a single request using aiohttp, to demonstrate how the keywords and work. We're going to use the Pokemon API as an example, so let's start by trying to get the data associated with the legendary 151st Pokemon, Mew.

Run the following Python code, and you should see the name "mew" printed to the terminal:

In this code, we're creating a coroutine called , which we are running with the asyncio event loop. In here we are opening an aiohttp client session, a single object that can be used for quite a number of individual requests and by default can make connections with up to 100 different servers at a time. With this session, we are making a request to the Pokemon API and then awaiting a response.

This keyword basically tells the Python interpreter that the coroutine we're defining should be run asynchronously with an event loop. The keyword passes control back to the event loop, suspending the execution of the surrounding coroutine and letting the event loop run other things until the result that is being "awaited" is returned.

Making a large number of requests

Making a single asynchronous HTTP request is great because we can let the event loop work on other tasks instead of blocking the entire thread while waiting for a response. But this functionality truly shines when trying to make a larger number of requests. Let's demonstrate this by performing the same request as before, but for all 150 of the original Pokemon.

Let's take the previous request code and put it in a loop, updating which Pokemon's data is being requested and using for each request:

This time, we're also measuring how much time the whole process takes. If you run this code in your Python shell, you should see something like the following printed to your terminal:

Result of 150 API calls using aiohttp

8 seconds seems pretty good for 150 requests, but we don't really have anything to compare it to. Let's try accomplishing the same thing synchronously using the requests library.

Comparing speed with synchronous requests

Requests was designed to be an HTTP library "for humans" so it has a very beautiful and simplistic API. I highly recommend it for any projects in which speed might not be of primary importance compared to developer-friendliness and easy to follow code.

To print the first 150 Pokemon as before, but using the requests library, run the following code:

You should see the same output with a different runtime:

Result of 150 API calls using requests

At nearly 29 seconds, this is significantly slower than the previous code. For each consecutive request, we have to wait for the previous step to finish before even beginning the process. It takes much longer because this code is waiting for 150 requests to finish sequentially

Utilizing asyncio for improved performance

So 8 seconds compared to 29 seconds is a huge jump in performance, but we can do even better using the tools that provides. In the original example, we are using after each individual HTTP request, which isn't quite ideal. It's still faster than the requests example because we are running everything in coroutines, but we can instead run all of these requests "concurrently" as asyncio tasks and then check the results at the end, using and .

If the code that actually makes the request is broken out into its own coroutine function, we can create a list of tasks, consisting of futures for each request. We can then unpack this list to a gather call, which runs them all together. When we this call to , we will get back an iterable for all of the futures that were passed in, maintaining their order in the list. This way we're only awaiting one time.

To see what happens when we implement this, run the following code:

This brings our time down to a mere 1.53 seconds for 150 HTTP requests! That is a vast improvement over even our initial async/await example. This example is completely non-blocking, so the total time to run all 150 requests is going to be roughly equal to the amount of time that the longest request took to run. The exact numbers will vary depending on your internet connection.

Result of 150 API calls using aiohttp and asyncio.gather

Concluding Thoughts

As you can see, using libraries like aiohttp to rethink the way you make HTTP requests can add a huge performance boost to your code and save a lot of time when making a large number of requests. By default, it is a bit more verbose than synchronous libraries like requests, but that is by design as the developers wanted to make performance a priority.

In this tutorial, we have only scratched the surface of what you can do with aiohttp and asyncio, but I hope that this has made starting your journey into the world of asynchronous Python a little easier.

I’m looking forward to seeing what you build. Feel free to reach out and share your experiences or ask any questions.

Sours: https://www.twilio.com/blog/asynchronous-http-requests-in-python-with-aiohttp
  1. Amd zero rpm
  2. Ninjago dragon sword
  3. Nyu director salary
  4. Pettigrew name origin
  5. Roissy indc

Creating a RESTful API with Python and aiohttp Image Creating a RESTful API with Python and aiohttp

This tutorial was built on top of Python 3.6

In this tutorial we’ll be building a very simple RESTful based API using aio-libs/aiohttp which is an asynchronous http client/server framework.

Getting Started with aiohttp

Before we go into how we can use aiohttp to create a simple RESTful API, it’s important to know exactly what the framework is and what it can do for us. To start with, it features excellent support of the HTTP protocol as well as for websockets which makes it ideal for working with popular websocket libraries such as Socket.io. If you are interested in seeing how to implement a simple client/server socketio based solution check out this tutorial: Python Socket.io with aiohttp Tutorial.

The key part of the aiohttp framework is that it works in an asynchronous manner, it can concurrently handle hundreds of requests per second without too much hassle. In comparison to frameworks such as flask, it’s incredibly performant.

Installing aiohttp

In order to install aiohttp you can run the following command:

Writing a Simple API

To get us started writing a simple API we are going to write a handler function; which will return a based response whenever it is called. We’ll then create an object by calling and then we’ll set up our app’s router and add a request endpoint that calls whenever is hit. Finally we call in order to kick off our newly defined API.

Testing our API

We can then run our new REST API by calling which should start our app on a.k.a on port by default.

When you navigate to you should see our being returned in the browser.

POST Requests and Query Parameters

Now that we’ve successfully defined a very basic, single API we can now start to build on top of this and start exposing different routes that use different verbs. Let’s create a simple request endpoint that takes in via a query parameter. We’ll want the final URL of this endpoint to look like so: . Let’s define the handler function now.

Once we have successfully defined this new handler function we will have to register it in our routes like so:

Try run your application now and send a request to and you should see the following output in the console:

You should also receive the same success json as well as a status.

Video Tutorial


Sours: https://tutorialedge.net/python/create-rest-api-python-aiohttp/
Web Scraping with AIOHTTP and Python

An intro to aiohttp

Python 3.5 added some new syntax that allows developers to create asynchronous applications and packages easier. One such package is aiohttp which is an HTTP client/server for asyncio. Basically it allows you to write asynchronous clients and servers. The aiohttp package also supports Server WebSockets and Client WebSockets. You can install aiohttp using pip:

pip install aiohttp

Now that we have aiohttp installed, let's take a look at one of their examples!


Fetching a Web Page

The documentation for aiohtpp has a fun example that shows how to grab a web page's HTML. Let's take a look at it and see how it works:

import aiohttp import asyncio import async_timeout async def fetch(session, url): with async_timeout.timeout(10): async with session.get(url) as response: return await response.text() async def main(loop): async with aiohttp.ClientSession(loop=loop) as session: html = await fetch(session, 'https://www.blog.pythonlibrary.org') print(html) loop = asyncio.get_event_loop() loop.run_until_complete(main(loop))

Here we just import aiohttp, Python's asyncio and async_timeout, which gives us the ability to timeout a coroutine. We create our event loop at the bottom of the code and call the main() function. It will create a ClientSession object that we pass to our fetch() function along with what URL to fetch. Finally in the fetch() function, we use set our timeout and attempt to get the URL's HTML. If everything works without timing out, you will see a bunch of text spewed into stdout.


Downloading Files with aiohttp

A fairly common task that developers will do is download files using threads or processes. We can download files using coroutines too! Let's find out how:

import aiohttp import asyncio import async_timeout import os async def download_coroutine(session, url): with async_timeout.timeout(10): async with session.get(url) as response: filename = os.path.basename(url) with open(filename, 'wb') as f_handle: while True: chunk = await response.content.read(1024) if not chunk: break f_handle.write(chunk) return await response.release() async def main(loop): urls = ["http://www.irs.gov/pub/irs-pdf/f1040.pdf", "http://www.irs.gov/pub/irs-pdf/f1040a.pdf", "http://www.irs.gov/pub/irs-pdf/f1040ez.pdf", "http://www.irs.gov/pub/irs-pdf/f1040es.pdf", "http://www.irs.gov/pub/irs-pdf/f1040sb.pdf"] async with aiohttp.ClientSession(loop=loop) as session: for url in urls: await download_coroutine(session, url) if __name__ == '__main__': loop = asyncio.get_event_loop() loop.run_until_complete(main(loop))

You will notice here that we import a couple of new items: aiohttp and async_timeout. The latter is a actually one of the aiohttp's dependencies and allows us to create a timeout context manager. Let's start at the bottom of the code and work our way up. In the bottom conditional statement, we start our asynchronous event loop and call our main function. In the main function, we create a ClientSession object that we pass on to our download coroutine function for each of the urls we want to download. In the download_coroutine, we create an async_timeout.timeout() context manager that basically creates a timer of X seconds. When the seconds run out, the context manager ends or times out. In this case, the timeout is 10 seconds. Next we call our session's get() method which gives us a response object. Now we get to the part that is a bit magical. When you use the content attribute of the response object, it returns an instance of aiohttp.StreamReader which allows us to download the file in chunks of whatever size we'd like. As we read the file, we write it out to local disk. Finally we call the response's release() method, which will finish the response processing.

According to aiohttp's documentation, because the response object was created in a context manager, it technically calls release() implicitly. But in Python, explicit is usually better and there is a note in the documentation that we shouldn't rely on the connection just going away, so I believe that it's better to just release it in this case.

There is one part that is still blocking here and that is the portion of the code that actually writes to disk. While we are writing the file, we are still blocking. There is another library called aiofiles that we could use to try and make the file writing asynchronous too We will take a look at that next.

Note: The section above came from one of my previous articles.


Using aiofiles For Asynchronous Writing

You will need to install aiofiles to make this work. Let's get that out of that way:

pip install aiofiles

Now that we have all the items we need, we can update our code! Note that this code only works in Python 3.6 or above.

import aiofiles import aiohttp import asyncio import async_timeout import os async def download_coroutine(session, url): with async_timeout.timeout(10): async with session.get(url) as response: filename = os.path.basename(url) async with aiofiles.open(filename, 'wb') as fd: while True: chunk = await response.content.read(1024) if not chunk: break await fd.write(chunk) return await response.release() async def main(loop, url): async with aiohttp.ClientSession(loop=loop) as session: await download_coroutine(session, url) if __name__ == '__main__': urls = ["http://www.irs.gov/pub/irs-pdf/f1040.pdf", "http://www.irs.gov/pub/irs-pdf/f1040a.pdf", "http://www.irs.gov/pub/irs-pdf/f1040ez.pdf", "http://www.irs.gov/pub/irs-pdf/f1040es.pdf", "http://www.irs.gov/pub/irs-pdf/f1040sb.pdf"] loop = asyncio.get_event_loop() loop.run_until_complete( asyncio.gather( *(main(loop, url) for url in urls) ) )

The only change is adding an import for aiofiles and then changing how we open the file. You will note that it is now

async with aiofiles.open(filename, 'wb') as fd:

And that we use await for the writing portion of the code:

await fd.write(chunk)

Other than that, the code is the same. There are some portability issues mentioned here that you should be aware of.


Wrapping Up

Now you should have some basic understanding of how to use aiohttp and aiofiles. The documentation for both projects is worth a look as this tutorial really only scratches the surface of what you can do with these libraries.


Related Reading

Sours: https://www.blog.pythonlibrary.org/2016/11/09/an-intro-to-aiohttp/

Tutorial aiohttp

Getting started

Let's start with basic folder structure:

  • project folder named . A root of the project. Run all commands from here.
  • application folder named inside of it
  • empty file . The place where web server will live

We need this nested so we can put config, tests and other related files next to it.

It looks like this:

polls <-- [current folder] └── aiohttpdemo_polls └── main.py

aiohttp server is built around :class:`aiohttp.web.Application` instance. It is used for registering startup/cleanup signals, connecting routes etc.

The following code creates an application:

# aiohttpdemo_polls/main.py from aiohttp import web app = web.Application() web.run_app(app)

Save it and start server by running:

$ python aiohttpdemo_polls/main.py ======== Running on http://0.0.0.0:8080 ======== (Press CTRL+C to quit)

Next, open the displayed link in a browser. It returns a error. To show something more meaningful than an error, let's create a route and a view.

Views

Let's start with the first views. Create the file and add the following to it:

# aiohttpdemo_polls/views.py from aiohttp import web async def index(request): return web.Response(text='Hello Aiohttp!')

This view is the simplest view possible in Aiohttp.

Now, we should create a route for this view. Put the following into . It is a good practice to separate views, routes, models etc. You'll have more of each file type, and it is nice to group them into different places:

# aiohttpdemo_polls/routes.py from views import index def setup_routes(app): app.router.add_get('/', index)

We should add a call to the function somewhere. The best place to do this is in :

# aiohttpdemo_polls/main.py from aiohttp import web from routes import setup_routes app = web.Application() setup_routes(app) web.run_app(app)

Start server again using . This time when we open the browser we see:

Hello Aiohttp!

Success! Now, your working directory should look like this:

. ├── .. └── polls └── aiohttpdemo_polls ├── main.py ├── routes.py └── views.py

Configuration files

Note

aiohttp is configuration agnostic. It means the library does not require any specific configuration approach, and it does not have built-in support for any config schema.

Please note these facts:

  1. 99% of servers have configuration files.

  2. Most products (except Python-based solutions like Django and Flask) do not store configs with source code.

    For example Nginx has its own configuration files stored by default under folder.

    MongoDB stores its config as .

  3. Config file validation is a good idea. Strong checks may prevent unnecessary errors during product deployment.

Thus, we suggest to use the following approach:

  1. Push configs as files ( or is also good but is preferred).
  2. Load config from a list of predefined locations, e.g. , .
  3. Keep the ability to override a config file by a command line parameter, e.g. .
  4. Apply strict validation checks to loaded dict. trafaret, colander or JSON schema are good candidates for such job.

One way to store your config is in folder at the same level as aiohttpdemo_polls. Create a folder and config file at desired location. E.g.:

. ├── .. └── polls <-- [BASE_DIR] │ ├── aiohttpdemo_polls │ ├── main.py │ ├── routes.py │ └── views.py │ └── config └── polls.yaml <-- [config file]

Create a file with meaningful option names:

# config/polls.yamlpostgres: database: aiohttpdemo_pollsuser: aiohttpdemo_userpassword: aiohttpdemo_passhost: localhostport: 5432minsize: 1maxsize: 5

Install package:

$ pip install pyyaml

Let's also create a separate file. It helps to leave clean and short:

# aiohttpdemo_polls/settings.py import pathlib import yaml BASE_DIR = pathlib.Path(__file__).parent.parent config_path = BASE_DIR / 'config' / 'polls.yaml' def get_config(path): with open(path) as f: config = yaml.safe_load(f) return config config = get_config(config_path)

Next, load the config into the application:

Now, try to run your app again. Make sure you are running it from :

$ python aiohttpdemo_polls/main.py ======== Running on http://0.0.0.0:8080 ======== (Press CTRL+C to quit)

For the moment nothing should have changed in application's behavior. But at least we know how to configure our application.

Database

Server

Here, we assume that you have running database and a user with write access. Refer to :ref:`aiohttp-demos-polls-preparations-database` for details.

Schema

We will use SQLAlchemy to describe database schema for two related models, and :

+---------------+ +---------------+ | question | | choice | +===============+ +===============+ | id | <---+ | id | +---------------+ | +---------------+ | question_text | | | choice_text | +---------------+ | +---------------+ | pub_date | | | votes | +---------------+ | +---------------+ +-------- | question_id | +---------------+

Create file with database schemas:

# aiohttpdemo_polls/db.py from sqlalchemy import ( MetaData, Table, Column, ForeignKey, Integer, String, Date ) meta = MetaData() question = Table( 'question', meta, Column('id', Integer, primary_key=True), Column('question_text', String(200), nullable=False), Column('pub_date', Date, nullable=False) ) choice = Table( 'choice', meta, Column('id', Integer, primary_key=True), Column('choice_text', String(200), nullable=False), Column('votes', Integer, server_default="0", nullable=False), Column('question_id', Integer, ForeignKey('question.id', ondelete='CASCADE')) )

Note

It is possible to configure tables in a declarative style like so:

classQuestion(Base): __tablename__='question'id=Column(Integer, primary_key=True) question_text=Column(String(200), nullable=False) pub_date=Column(Date, nullable=False)

But it doesn't give much benefits later on. SQLAlchemy ORM doesn't work in asynchronous style and as a result doesn't support related ORM expressions such as or .

You still can make queries after some code modifications:

fromsqlalchemy.sqlimportselectresult=awaitconn.execute(select([Question]))

instead of

result=awaitconn.execute(question.select())

But it is not as easy to deal with as update/delete queries.

Now we need to create tables in database as it was described with sqlalchemy. Helper script can do that for you. Create a new file in project's root:

# polls/init_db.py from sqlalchemy import create_engine, MetaData from aiohttpdemo_polls.settings import config from aiohttpdemo_polls.db import question, choice DSN = "postgresql://{user}:{password}@{host}:{port}/{database}" def create_tables(engine): meta = MetaData() meta.create_all(bind=engine, tables=[question, choice]) def sample_data(engine): conn = engine.connect() conn.execute(question.insert(), [ {'question_text': 'What\'s new?', 'pub_date': '2015-12-15 17:17:49.629+02'} ]) conn.execute(choice.insert(), [ {'choice_text': 'Not much', 'votes': 0, 'question_id': 1}, {'choice_text': 'The sky', 'votes': 0, 'question_id': 1}, {'choice_text': 'Just hacking again', 'votes': 0, 'question_id': 1}, ]) conn.close() if __name__ == '__main__': db_url = DSN.format(**config['postgres']) engine = create_engine(db_url) create_tables(engine) sample_data(engine)

Install the package (it will pull alongside) to interact with the database, and run the script:

$ pip install aiopg[sa] $ python init_db.py

Note

At this point we are not using any async features of the package. For this reason, you could have installed package. Though since we are using sqlalchemy, we also could switch the type of database server.

Now there should be one record for question with related choice options stored in corresponding tables in the database.

Use , or any other tool you like to check database contents:

$ psql -U postgres -h localhost -p 5432 -d aiohttpdemo_polls aiohttpdemo_polls=# select * from question; id | question_text | pub_date ----+---------------+------------ 1 | What's new? | 2015-12-15 (1 row)

Doing things at startup and shutdown

Sometimes it is necessary to configure some component's setup and tear down. For a database this would be the creation of a connection or connection pool and closing it afterwards.

Pieces of code below belong to and files. Complete files will be shown shortly after.

Creating connection engine

For making DB queries we need an engine instance. Assuming is a :class:`dict` with the configuration info for a Postgres connection, this could be done by the following async generator function:

.. literalinclude:: ../demos/polls/aiohttpdemo_polls/db.py :pyobject: pg_context

Add the code to file.

The best place for connecting to the DB is using the :attr:`~aiohtp.web.Application.cleanup_ctx` signal:

app.cleanup_ctx.append(pg_context)

On startup, the code is run until the . When the application is shutdown the code will resume and close the DB connection.

Complete files with changes

Since we now have database connection on start - let's use it! Modify index view:

# aiohttpdemo_polls/views.pyfromaiohttpimportwebimportdbasyncdefindex(request): asyncwithrequest.app['db'].acquire() asconn: cursor=awaitconn.execute(db.question.select()) records=awaitcursor.fetchall() questions= [dict(q) forqinrecords] returnweb.Response(text=str(questions))

Run server and you should get list of available questions (one record at the moment) with all fields.

Templates

For setting up the template engine, we install the library first:

$ pip install aiohttp_jinja2

After installing, setup the library:

As you can see from setup above - templates should be placed at folder.

Let's create simple template and modify index view to use it:

<!--aiohttpdemo_polls/templates/index.html-->{%settitle = "Main"%}{%ifquestions%} <ul> {%forquestioninquestions%} <li>{{ question.question_text }}</li> {%endfor%} </ul> {%else%} <p>No questions are available.</p> {%endif%}

Templates are a very convenient way for web page writing. If we return a dict with page content, the decorator processes the dict using the jinja2 template renderer.

Run the server and you should see a question decorated in html list element.

Let's add more views:

.. literalinclude:: ../demos/polls/aiohttpdemo_polls/views.py :pyobject: poll

Static files

Any web site has static files such as: images, JavaScript sources, CSS files

The best way to handle static files in production is by setting up a reverse proxy like NGINX or using CDN services.

During development, handling static files using the aiohttp server is very convenient.

Fortunately, this can be done easily by a single call:

.. literalinclude:: ../demos/polls/aiohttpdemo_polls/routes.py :pyobject: setup_static_routes

where is the path to the root folder.

Middlewares

Middlewares are stacked around every web-handler. They are called before the handler for a pre-processing request. After getting a response back, they are used for post-processing the given response.

A common use of middlewares is to implement custom error pages. Example from :ref:`aiohttp-web-middlewares` documentation will render 404 errors using a JSON response, as might be appropriate for a REST service.

Here we'll create a little bit more complex middleware custom display pages for 404 Not Found and 500 Internal Error.

Every middleware should accept two parameters, a request and a handler, and return the response. Middleware itself is a coroutine that can modify either request or response:

Now, create a new file:

.. literalinclude:: ../demos/polls/aiohttpdemo_polls/middlewares.py

As you can see, we do nothing before the web handler. In the case of an , we use the Jinja2 template renderer based on after the request was handled. For other exceptions, we log the error and render our 500 template. Without the function, the same task would take us many more statements.

We have registered middleware in by adding it to .

Now, add a step to the main file:

Run the app again. To test, try an invalid url.

Sours: https://github.com/aio-libs/aiohttp-demos/blob/master/docs/tutorial.rst
Asyncio: Understanding Async / Await in Python

Welcome to AIOHTTP¶

Asynchronous HTTP Client/Server for asyncio and Python.

Current version is 3.7.4.post0.

Library Installation¶

You may want to install optionalcchardet library as faster replacement for chardet:

For speeding up DNS resolving by client API you may install aiodns as well. This option is highly recommended:

Installing speedups altogether¶

The following will get you along with chardet, aiodns and in one bundle. No need to type separate commands anymore!

$ pip install aiohttp[speedups]

Getting Started¶

Client example¶

importaiohttpimportasyncioasyncdefmain():asyncwithaiohttp.ClientSession()assession:asyncwithsession.get('http://python.org')asresponse:print("Status:",response.status)print("Content-type:",response.headers['content-type'])html=awaitresponse.text()print("Body:",html[:15],"...")loop=asyncio.get_event_loop()loop.run_until_complete(main())

This prints:

Status: 200 Content-type: text/html; charset=utf-8 Body: <!doctype html> ...

Coming from requests ? Read why we need so many lines.

Server example:¶

fromaiohttpimportwebasyncdefhandle(request):name=request.match_info.get('name',"Anonymous")text="Hello, "+namereturnweb.Response(text=text)app=web.Application()app.add_routes([web.get('/',handle),web.get('/{name}',handle)])if__name__=='__main__':web.run_app(app)

For more information please visit Client and Server pages.

Source code¶

The project is hosted on GitHub

Please feel free to file an issue on the bug tracker if you have found a bug or have some suggestion in order to improve the library.

The library uses Azure Pipelines for Continuous Integration.

Dependencies¶

  • Python 3.6+

  • async_timeout

  • attrs

  • chardet

  • multidict

  • yarl

  • Optionalcchardet as faster replacement for chardet.

    Install it explicitly via:

  • Optionalaiodns for fast DNS resolving. The library is highly recommended.

Authors and License¶

The package is written mostly by Nikolay Kim and Andrew Svetlov.

It’s Apache 2 licensed and freely available.

Feel free to improve this package and send a pull request to GitHub.

Policy for Backward Incompatible Changes¶

aiohttp keeps backward compatibility.

After deprecating some Public API (method, class, function argument, etc.) the library guaranties the usage of deprecated API is still allowed at least for a year and half after publishing new release with deprecation.

All deprecations are reflected in documentation and raises .

Sometimes we are forced to break the own rule for sake of very strong reason. Most likely the reason is a critical bug which cannot be solved without major API change, but we are working hard for keeping these changes as rare as possible.

Sours: https://docs.aiohttp.org/

Now discussing:

It seemed that even fiddling with these sticks gave her almost as much pleasure as shooting. It is better for you to cherish and groom your instrument. And then I brought a bottle of shampoos and cakes: - We must, my love, celebrate the.



537 538 539 540 541