Introduction to Python coroutines

There is much more to yield!

Noam Elfanbaum
noamelf.com
@noamelf

Introduction

  • In 1963 Melvine Conway (not the same guy from the Game of Life) came out with a paradaigm that will allow him to decouple COBOL programs into seperate units (Design of a Separable Transition-Diagram Compiler). Instead of a routine that pass control to a subroutine and then gets it back, you have co-routines that passes control simultanoiusly from one another.

Subroutines are special cases of... coroutines. – Donald Knuth

  • Coroutines paradigm was introduced to Python 2.5 in PEP 342 with the motivation of expanding the language expressivness as Guido and Phillip write in the PEP motivation section:

    Coroutines are a natural way of expressing many algorithms, such as simulations, games, asynchronous I/O, and other forms of event- driven programming or co-operative multitasking.

Basics

Each componenet is one of the building blocks of the one underneath:

  • Iterators
  • Generators
  • Coroutines

Iterators

Iterator objects represents a sequence of data. The iterator specification (protocol) includes 2 methods each iterator object must implement:

  • __iter__ - return the object itself.
  • __next__ - returns the next object in the sequence, raises StopIteration if empty.
In [30]:
# A class implementing the iterator protocol
class iter_range(object):
    def __init__(self, n):
        self.i = 0
        self.n = n

    def __iter__(self):
        return self

    def  __next__ (self):
        if self.i < self.n:
            i = self.i
            self.i += 1
            return i
        else:
            raise StopIteration()
In [31]:
it = iter_range(3)
for i in it:
    print(i, end=' ')
0 1 2 

Generators

Generators helps us generate iterators easily. A generator is essentially a functions that contains a yield statement in it's body.

In [32]:
def gen_range(n):
    i = 0
    while i < n:
        yield i
        i += 1
In [33]:
it = gen_range(3)
for i in it:
    print(i, end=' ')
0 1 2 

Coroutines

Coroutines are very similar to generators but they utilize the yield statement to consume values.

In [34]:
def grep(pattern):
 print('Looking for {}'.format(pattern))
 while True:
     line = yield
     if pattern in line:
         print(line)
In [35]:
g = grep('python')
next(g)
Looking for python
In [36]:
g.send('Yeah, but no, but yeah, but no')
g.send('A series of tubes')
g.send("python coroutines rock!")
python coroutines rock!

Coroutine initialization

A coroutine decorater to remove the need for calling next each time.

In [37]:
def coroutine(func):
    def start(*args,**kwargs):
        cr = func(*args,**kwargs)
        next(cr)
        return cr
    return start

close and throw

  • When close is called a coroutine, a GeneratorExit exception is thrown, and the coroutine will end (the fact the exception is caught doesn't matter, the coroutine will exit anyhow).
  • Same goes for throw, just that you can decide which exeption will be thrown and you can return a value (see examples).
In [38]:
# close() example
@coroutine
def add_up():
    s = 0
    try:
        while True:
            x = yield
            print(x, end=' ')
            s += x
    except GeneratorExit:
        print('\nThe sum is {}. Cleaning up.'.format(s))
In [39]:
adder_gen = add_up()
for i in range(5):
    adder_gen.send(i)
adder_gen.close()
0 1 2 3 4 
The sum is 10. Cleaning up.
In [40]:
# throw() example
@coroutine
def throw_me_an_exception():
    try:
        while True:
            x = yield
    except InterruptedError as e:
        print(e)
        yield 'got it' 
In [41]:
thrower_gen = throw_me_an_exception()
answer = thrower_gen.throw(InterruptedError, 'stop it')
print(answer)

#next(thrower_gen)
stop it
got it

Objects vs Coroutines

It seems that there is a great similarity between regular objects and coroutines: they both has data members that can hold states. Then why use coroutines? Let's see David Beasely's grep benchmark comparison:

In [42]:
# Benchmark authored by David Beasely - http://www.dabeaz.com/coroutines/index.html

class GrepHandler(object):
    def __init__(self, pattern, target):
        self.pattern = pattern
        self.target = target

    def send(self, line):
        if self.pattern in line:
            self.target.send(line)


@coroutine
def grep(pattern, target):
    while True:
        line = yield
        if pattern in line:
            target.send(line)


# A null-sink to send data
@coroutine
def null():
    while True: item = yield
   
line = 'python is nice'
p1 = grep('python', null())  # Coroutine
p2 = GrepHandler('python', null())  # Object

%timeit -n 1000 p1.send(line)
%timeit -n 1000 p2.send(line)
1000 loops, best of 3: 423 ns per loop
1000 loops, best of 3: 465 ns per loop

So by using coroutines we get:

  • Elegance (agruable).
  • Speed.
  • Job security (only you understand the code :) )

Use cases

As PEP 342 mentions and Knuth & Ruskey decribes thouroughly in Deconstructing Coroutines, coroutines can supply an elegant solution to a varaity of problems, in fact Python's asycio package is based on them being called by an event loop (but that's for another talk).

Use case 1 - Pipes

  • Pipes allows us to proccess large amounts of data without worrying about memory footprint and following unexpected crushes.
  • Since we each chunk of data finishes the cycle as it is read, even if some mischief happens, the proccess does not need restart from the top.
  • Using coroutines we can create pipes that are produced from small and modular units (hence unittests friendly).
In [43]:
# An illustration of coroutines 'branching' ability.

from IPython.display import Image,display_png
display_png(Image('images/branchy.png'))
In [48]:
# A sink.  A coroutine that receives data
@coroutine
def printer():
    while True:
         line = yield
         print(line)

# A filter.
@coroutine
def grep(pattern,target):
    while True:
        line = yield
        if pattern in line:
            target.send(line) 

# Broadcast a stream onto multiple targets
@coroutine
def broadcast(*targets):
    while True:
        item = yield
        for target in targets:
            target.send(item)

def producer(broadcast):
    with open('log') as f:
        for line in f:
            broadcaster.send(line)
            
if __name__ == '__main__':
    p = printer()
    producer(broadcast(grep('startup',p), grep('font',p), grep('libjs',p)))
    
2015-05-01 11:13:38 startup archives unpack

2015-05-01 11:13:45 install fonts-font-awesome:all <none> 4.2.0~dfsg-1

2015-05-01 11:13:45 status half-installed fonts-font-awesome:all 4.2.0~dfsg-1

2015-05-01 11:13:45 status triggers-pending fontconfig:amd64 2.11.1-0ubuntu6

2015-05-01 11:13:46 status unpacked fonts-font-awesome:all 4.2.0~dfsg-1

2015-05-01 11:13:46 status unpacked fonts-font-awesome:all 4.2.0~dfsg-1

2015-05-01 11:13:46 install fonts-mathjax:all <none> 2.5.1-1

2015-05-01 11:13:46 status half-installed fonts-mathjax:all 2.5.1-1

2015-05-01 11:13:46 status unpacked fonts-mathjax:all 2.5.1-1

2015-05-01 11:13:46 status unpacked fonts-mathjax:all 2.5.1-1

2015-05-01 11:13:47 install libjs-highlight.js:all <none> 8.2+ds-4

2015-05-01 11:13:47 status half-installed libjs-highlight.js:all 8.2+ds-4

2015-05-01 11:13:47 status unpacked libjs-highlight.js:all 8.2+ds-4

2015-05-01 11:13:47 status unpacked libjs-highlight.js:all 8.2+ds-4

2015-05-01 11:13:47 install libjs-highlight:all <none> 8.2+ds-4

2015-05-01 11:13:47 status half-installed libjs-highlight:all 8.2+ds-4

2015-05-01 11:13:48 status unpacked libjs-highlight:all 8.2+ds-4

2015-05-01 11:13:48 status unpacked libjs-highlight:all 8.2+ds-4

2015-05-01 11:13:48 install libjs-jquery:all <none> 1.7.2+dfsg-3ubuntu2

2015-05-01 11:13:48 status half-installed libjs-jquery:all 1.7.2+dfsg-3ubuntu2

2015-05-01 11:13:49 status unpacked libjs-jquery:all 1.7.2+dfsg-3ubuntu2

2015-05-01 11:13:49 status unpacked libjs-jquery:all 1.7.2+dfsg-3ubuntu2

2015-05-01 11:13:49 install libjs-jquery-ui:all <none> 1.10.1+dfsg-1

2015-05-01 11:13:49 status half-installed libjs-jquery-ui:all 1.10.1+dfsg-1

2015-05-01 11:13:52 status unpacked libjs-jquery-ui:all 1.10.1+dfsg-1

2015-05-01 11:13:53 status unpacked libjs-jquery-ui:all 1.10.1+dfsg-1

2015-05-01 11:13:53 install libjs-marked:all <none> 0.3.2+dfsg-1

2015-05-01 11:13:53 status half-installed libjs-marked:all 0.3.2+dfsg-1

2015-05-01 11:13:54 status unpacked libjs-marked:all 0.3.2+dfsg-1

2015-05-01 11:13:55 status unpacked libjs-marked:all 0.3.2+dfsg-1

2015-05-01 11:13:55 install libjs-mathjax:all <none> 2.5.1-1

2015-05-01 11:13:55 status half-installed libjs-mathjax:all 2.5.1-1

2015-05-01 11:13:59 status unpacked libjs-mathjax:all 2.5.1-1

2015-05-01 11:13:59 status unpacked libjs-mathjax:all 2.5.1-1

2015-05-01 11:13:59 install libjs-underscore:all <none> 1.7.0~dfsg-1ubuntu1

2015-05-01 11:13:59 status half-installed libjs-underscore:all 1.7.0~dfsg-1ubuntu1

2015-05-01 11:13:59 status unpacked libjs-underscore:all 1.7.0~dfsg-1ubuntu1

2015-05-01 11:13:59 status unpacked libjs-underscore:all 1.7.0~dfsg-1ubuntu1

2015-05-01 11:14:09 trigproc fontconfig:amd64 2.11.1-0ubuntu6 <none>

2015-05-01 11:14:09 status half-configured fontconfig:amd64 2.11.1-0ubuntu6

2015-05-01 11:14:09 status installed fontconfig:amd64 2.11.1-0ubuntu6

2015-05-01 11:14:14 startup packages configure

2015-05-01 11:14:14 configure fonts-font-awesome:all 4.2.0~dfsg-1 <none>

2015-05-01 11:14:14 status unpacked fonts-font-awesome:all 4.2.0~dfsg-1

2015-05-01 11:14:14 status half-configured fonts-font-awesome:all 4.2.0~dfsg-1

2015-05-01 11:14:14 status installed fonts-font-awesome:all 4.2.0~dfsg-1

2015-05-01 11:14:14 configure fonts-mathjax:all 2.5.1-1 <none>

2015-05-01 11:14:14 status unpacked fonts-mathjax:all 2.5.1-1

2015-05-01 11:14:14 status half-configured fonts-mathjax:all 2.5.1-1

2015-05-01 11:14:14 status installed fonts-mathjax:all 2.5.1-1

2015-05-01 11:14:14 configure libjs-highlight.js:all 8.2+ds-4 <none>

2015-05-01 11:14:14 status unpacked libjs-highlight.js:all 8.2+ds-4

2015-05-01 11:14:14 status half-configured libjs-highlight.js:all 8.2+ds-4

2015-05-01 11:14:14 status installed libjs-highlight.js:all 8.2+ds-4

2015-05-01 11:14:14 configure libjs-highlight:all 8.2+ds-4 <none>

2015-05-01 11:14:14 status unpacked libjs-highlight:all 8.2+ds-4

2015-05-01 11:14:14 status half-configured libjs-highlight:all 8.2+ds-4

2015-05-01 11:14:15 status installed libjs-highlight:all 8.2+ds-4

2015-05-01 11:14:15 configure libjs-jquery:all 1.7.2+dfsg-3ubuntu2 <none>

2015-05-01 11:14:15 status unpacked libjs-jquery:all 1.7.2+dfsg-3ubuntu2

2015-05-01 11:14:15 status half-configured libjs-jquery:all 1.7.2+dfsg-3ubuntu2

2015-05-01 11:14:15 status installed libjs-jquery:all 1.7.2+dfsg-3ubuntu2

2015-05-01 11:14:15 configure libjs-jquery-ui:all 1.10.1+dfsg-1 <none>

2015-05-01 11:14:15 status unpacked libjs-jquery-ui:all 1.10.1+dfsg-1

2015-05-01 11:14:15 status half-configured libjs-jquery-ui:all 1.10.1+dfsg-1

2015-05-01 11:14:15 status installed libjs-jquery-ui:all 1.10.1+dfsg-1

2015-05-01 11:14:15 configure libjs-marked:all 0.3.2+dfsg-1 <none>

2015-05-01 11:14:15 status unpacked libjs-marked:all 0.3.2+dfsg-1

2015-05-01 11:14:15 status half-configured libjs-marked:all 0.3.2+dfsg-1

2015-05-01 11:14:15 status installed libjs-marked:all 0.3.2+dfsg-1

2015-05-01 11:14:15 configure libjs-mathjax:all 2.5.1-1 <none>

2015-05-01 11:14:15 status unpacked libjs-mathjax:all 2.5.1-1

2015-05-01 11:14:16 status half-configured libjs-mathjax:all 2.5.1-1

2015-05-01 11:14:16 status installed libjs-mathjax:all 2.5.1-1

2015-05-01 11:14:16 configure libjs-underscore:all 1.7.0~dfsg-1ubuntu1 <none>

2015-05-01 11:14:16 status unpacked libjs-underscore:all 1.7.0~dfsg-1ubuntu1

2015-05-01 11:14:16 status half-configured libjs-underscore:all 1.7.0~dfsg-1ubuntu1

2015-05-01 11:14:16 status installed libjs-underscore:all 1.7.0~dfsg-1ubuntu1

2015-05-01 11:14:21 startup packages configure

2015-05-04 11:58:08 startup archives unpack

2015-05-04 11:58:30 startup packages configure

2015-05-04 11:58:33 startup packages configure

2015-05-04 13:30:49 startup archives unpack

2015-05-04 13:31:05 startup packages configure

2015-05-04 13:31:09 startup packages configure

2015-05-04 14:36:24 startup archives unpack

2015-05-04 14:36:52 startup packages configure

2015-05-04 14:37:06 startup packages configure

2015-05-04 14:37:07 startup archives install

Pipes example #2

In this example, we need fetch data from furniture.com. As with any pipe we can abstract the steps taken to 3:

  1. Producer - API request to furniture.com.
  2. Filter - filtering and formating the data to match our needs.
  3. Sink/Consumers - Update our DB and indexer.