Introduction¶
- In 1963 Melvine Conway (not the same guy from the Game of Life) came out with a paradaigm that will allow him to decouple COBOL programs into seperate units (Design of a Separable Transition-Diagram Compiler). Instead of a routine that pass control to a subroutine and then gets it back, you have co-routines that passes control simultanoiusly from one another.
Subroutines are special cases of... coroutines. – Donald Knuth
- Coroutines paradigm was introduced to Python 2.5 in PEP 342 with the motivation of expanding the language expressivness as Guido and Phillip write in the PEP motivation section:
Coroutines are a natural way of expressing many algorithms, such as simulations, games, asynchronous I/O, and other forms of event- driven programming or co-operative multitasking.
Basics¶
Each componenet is one of the building blocks of the one underneath:
- Iterators
- Generators
- Coroutines
Iterators¶
Iterator objects represents a sequence of data. The iterator specification (protocol) includes 2 methods each iterator object must implement:
__iter__- return the object itself.__next__- returns the next object in the sequence, raisesStopIterationif empty.
# A class implementing the iterator protocol
class iter_range(object):
def __init__(self, n):
self.i = 0
self.n = n
def __iter__(self):
return self
def __next__ (self):
if self.i < self.n:
i = self.i
self.i += 1
return i
else:
raise StopIteration()
it = iter_range(3)
for i in it:
print(i, end=' ')
Generators¶
Generators helps us generate iterators easily. A generator is essentially a functions that contains a yield statement in it's body.
def gen_range(n):
i = 0
while i < n:
yield i
i += 1
it = gen_range(3)
for i in it:
print(i, end=' ')
Coroutines¶
Coroutines are very similar to generators but they utilize the yield statement to consume values.
def grep(pattern):
print('Looking for {}'.format(pattern))
while True:
line = yield
if pattern in line:
print(line)
g = grep('python')
next(g)
g.send('Yeah, but no, but yeah, but no')
g.send('A series of tubes')
g.send("python coroutines rock!")
Coroutine initialization¶
A coroutine decorater to remove the need for calling next each time.
def coroutine(func):
def start(*args,**kwargs):
cr = func(*args,**kwargs)
next(cr)
return cr
return start
close and throw¶
- When
closeis called a coroutine, aGeneratorExitexception is thrown, and the coroutine will end (the fact the exception is caught doesn't matter, the coroutine will exit anyhow). - Same goes for
throw, just that you can decide which exeption will be thrown and you can return a value (see examples).
# close() example
@coroutine
def add_up():
s = 0
try:
while True:
x = yield
print(x, end=' ')
s += x
except GeneratorExit:
print('\nThe sum is {}. Cleaning up.'.format(s))
adder_gen = add_up()
for i in range(5):
adder_gen.send(i)
adder_gen.close()
# throw() example
@coroutine
def throw_me_an_exception():
try:
while True:
x = yield
except InterruptedError as e:
print(e)
yield 'got it'
thrower_gen = throw_me_an_exception()
answer = thrower_gen.throw(InterruptedError, 'stop it')
print(answer)
#next(thrower_gen)
Objects vs Coroutines¶
It seems that there is a great similarity between regular objects and coroutines: they both has data members that can hold states. Then why use coroutines? Let's see David Beasely's grep benchmark comparison:
# Benchmark authored by David Beasely - http://www.dabeaz.com/coroutines/index.html
class GrepHandler(object):
def __init__(self, pattern, target):
self.pattern = pattern
self.target = target
def send(self, line):
if self.pattern in line:
self.target.send(line)
@coroutine
def grep(pattern, target):
while True:
line = yield
if pattern in line:
target.send(line)
# A null-sink to send data
@coroutine
def null():
while True: item = yield
line = 'python is nice'
p1 = grep('python', null()) # Coroutine
p2 = GrepHandler('python', null()) # Object
%timeit -n 1000 p1.send(line)
%timeit -n 1000 p2.send(line)
So by using coroutines we get:
- Elegance (agruable).
- Speed.
- Job security (only you understand the code :) )
Use cases¶
As PEP 342 mentions and Knuth & Ruskey decribes thouroughly in Deconstructing Coroutines, coroutines can supply an elegant solution to a varaity of problems, in fact Python's asycio package is based on them being called by an event loop (but that's for another talk).
Use case 1 - Pipes¶
- Pipes allows us to proccess large amounts of data without worrying about memory footprint and following unexpected crushes.
- Since we each chunk of data finishes the cycle as it is read, even if some mischief happens, the proccess does not need restart from the top.
- Using coroutines we can create pipes that are produced from small and modular units (hence unittests friendly).
# An illustration of coroutines 'branching' ability.
from IPython.display import Image,display_png
display_png(Image('images/branchy.png'))
# A sink. A coroutine that receives data
@coroutine
def printer():
while True:
line = yield
print(line)
# A filter.
@coroutine
def grep(pattern,target):
while True:
line = yield
if pattern in line:
target.send(line)
# Broadcast a stream onto multiple targets
@coroutine
def broadcast(*targets):
while True:
item = yield
for target in targets:
target.send(item)
def producer(broadcast):
with open('log') as f:
for line in f:
broadcaster.send(line)
if __name__ == '__main__':
p = printer()
producer(broadcast(grep('startup',p), grep('font',p), grep('libjs',p)))
Pipes example #2¶
In this example, we need fetch data from furniture.com. As with any pipe we can abstract the steps taken to 3:
- Producer - API request to furniture.com.
- Filter - filtering and formating the data to match our needs.
- Sink/Consumers - Update our DB and indexer.