Python’s asynchronous programming functionality, or async for short, allows you to write programs that get more work done by not waiting for independent tasks to finish. The asyncio library included with Python gives you the tools to use async for processing disk or network I/O without making everything else wait.
asyncio provides two kinds of APIs for dealing with asynchronous operations: high-level and low-level. The high-level APIs are the most generally useful, and they’re applicable to the widest variety of applications. The low-level APIs are powerful, but also complex, and used less frequently.
We’ll concentrate on the high-level APIs in this article. In the sections below, we’ll walk through the most commonly used high-level APIs in asyncio, and show how they can be used for common operations involving asynchronous tasks.
If you’re completely new to async in Python, or you could use a refresher on how it works, read my introduction to Python async before diving in here.
Run coroutines and tasks in Python
Naturally, the most common use for asyncio is to run the asynchronous parts of your Python script. This means learning to work with coroutines and tasks.
Python’s async components, including coroutines and tasks, can only be used with other async components, and not with conventional synchronous Python, so you need asyncio to bridge the gap. To do this, you use the asyncio.run function:
import asyncio
async def main():
print ("Waiting 5 seconds. ")
for _ in range(5):
await asyncio.sleep(1)
print (".")
print ("Finished waiting.")
asyncio.run(main())
This runs main(), along with any coroutines main() fires off, and waits for a result to return.
As a general rule, a Python program should have only one .run() statement, just as a Python program should have only one main() function. Async, if used carelessly, can make the control flow of a program hard to read. Having a single entry point to a program’s async code keeps things from getting hairy.
Async functions can also be scheduled as tasks, or objects that wrap coroutines and help run them.
async def my_task():
do_something()
task = asyncio.create_task(my_task())
my_task() is then run in the event loop, with its results stored in task.
If you have only one task you want to get results from, you can use asyncio.wait_for(task) to wait for the task to finish, then use task.result() to retrieve its result. But if you’ve scheduled a number of tasks to execute and you want to wait for all of them to finish, use asyncio.wait([task1, task2]) to gather the results. (Note that you can set a timeout for the operations if you don’t want them to run past a certain length of time.)
Manage an async event loop in Python
Another common use for asyncio is to manage the async event loop. The event loop is an object that runs async functions and callbacks; it’s created automatically when you use asyncio.run(). You generally want to use only one async event loop per program, again to keep things manageable.
If you’re writing more advanced software, such as a server, you’ll need lower-level access to the event loop. To that end, you can “lift the hood” and work directly with the event loop’s internals. But for simple jobs you won’t need to.
Read and write data with streams in Python
The best scenarios for async are long-running network operations, where the application may block waiting for some other resource to return a result. To that end, asyncio offers streams, which are high-level mechanisms for performing network I/O. This includes acting as a server for network requests.
asyncio uses two classes, StreamReader and StreamWriter, to read and write from the network at a high level. If you want to read from the network, you would use asyncio.open_connection() to open the connection. That function returns a tuple of StreamReader and StreamWriter objects, and you would use .read() and .write() methods on each to communicate.
To receive connections from remote hosts, use asyncio.start_server(). The asyncio.start_server() function takes as an argument a callback function, client_connected_cb, which is called whenever it receives a request. That callback function takes instances of StreamReader and StreamWriter as arguments, so you can handle the read/write logic for the server. (See here for an example of a simple HTTP server that uses the asyncio-driven aiohttp library.)
Synchronize tasks in Python
Asynchronous tasks tend to run in isolation, but sometimes you will want them to communicate with each other. asyncio provides queues and several other mechanisms for synchronizing between tasks:
-
Queues:
asyncioqueues allow asynchronous functions to line up Python objects to be consumed by other async functions — for instance, to distribute workloads between different kinds of functions based on their behaviors. -
Synchronization primitives: Locks, events, conditions, and semaphores in
asynciowork like their conventional Python counterparts.
One thing to keep in mind about all of these methods is that they’re not thread-safe. This isn’t an issue for async tasks running in the same event loop. But if you’re trying to share information with tasks in a different event loop, OS thread, or process, you’ll need to use the threading module and its objects to do that.
Further, if you want to launch coroutines across thread boundaries, use the asyncio.run_coroutine_threadsafe() function, and pass the event loop to use with it as a parameter.
Pause a coroutine in Python
Another common use of asyncio, and an under-discussed one, is waiting for some arbitrary length of time inside a coroutine. You can’t use time.sleep() for this, or you’ll block the entire program. Instead, use asyncio.sleep(), which allows other coroutines to continue running.
Use lower-level async in Python
Finally, if you think that the app you’re building may require asyncio’s lower-level components, take a look around before you start coding: There’s a good chance someone has already built an async-powered Python library that does what you need.
For instance, if you need async DNS querying, check the aiodns library, and for async SSH sessions, there’s asyncSSH. Search PyPI by the keyword “async” (plus other task-related keywords), or check the hand-curated Awesome Asyncio list for ideas.
This story, "How to use asyncio in Python" was originally published by
Gloss