DeferredGenerator

There are a variety of situations that require handling events asynchronously e.g., performing network or disk I/O or handling user interaction. The typical solutions are threading or using an event loop. Threading introduces locking problems like deadlock and livelock. Event loops avoid locking problems, but require awkward structuring of code. An example of this is the Deferred mechanism of the Python Twisted library. DeferredGenerator builds on Deferred to provide an asynchronous event handling mechanism that is easier to use than Deferred while still avoiding locking issues. This page describes the motivation and usage of DeferredGenerator and assumes that readers are already familiar with Twisted's Deferred mechanism.

There are actually two DeferredGenerator mechanisms: the one in the standard twisted library and the Tycoon version. This page is about the Tycoon version. The basic idea in both cases is the same: turn a normal function into a generator by wrapping it using the deferredGenerator function and use yield to indicate breaks in the control flow. The Tycoon version is less verbose and uses Twisted's Failure mechanism to propagte errors rather than the Python exception mechanism.

Here are the things to keep in mind when using Tycoon's DeferredGenerator:

1. Be sure to the wrap the function with deferredGenerator.
2. If you want to do error handling, add an errback to the deferred returned from the function whose error you want to handle. This exactly like the standard error handling process for normal deferreds.
3. If the errback does not return a failure, then execution will resume after the yield of the deferred that contained the failure; otherwise, execution of this function will terminate and the failure will be the return result.
4. All yields except the last one should yield a deferred.
5. The last yield should yield the return value of this function.
6. All explicit and implicit return statements should not return anything.
7. Once wrapped, the function behaves exactly like any other function that returns a deferred, so callers of the function do not need to be (and in fact, should not be) aware of how this function is implemented.

This is an example of code using the Tycoon DeferredGenerator:

   def f(a, b, c):
       d = operationReturningDeferred()
       d.addErrback(g)
       yield d
       print d.result, a, b, c
       d = anotherOperationReturningDeferred()
       yield d
       print d.result, a, b, c
       d = lastOperationReturningDeferred()
       yield d
       print d.result, a, b, c
       yield 6

   f = deferredGenerator(f)

Note the following:

1. f is wrapped using deferredGenerator.
2. The deferred returned from operationReturningDeferred has an errback added to it to handle errors that occurred in operationReturningDeferred.
3. If operationReturningDeferred returns a Failure, then g will be called. If g returns a Failure, then f returns with that failure. Otherwise, d.result contains the return value of g and execution resumes after the first yield.
4. Only the last yield returns something (6) other than a deferred.
5. The implicit return does not return anything.
6. f can be called like any other function that returns a deferred.

Contrast this with the same code using only deferreds:

   def f(a, b, c):
       d = operationReturningDeferred()
       d.addErrback(g)
       d.addCallback(f1, a, b, c)
       return d

   def f1(result, a, b, c):
       print result, a, b, c
       d = anotherOperationReturningDeferred()
       d.addCallback(f2, a, b, c)
       return d

   def f2(result, a, b, c):
       print result, a, b, c
       d = lastOperationReturningDeferred()
       d.addCallback(f3, a, b, c)
       return d

   def f3(result, a, b, c):
       print result, a, b, c
       return 6

This version is not only longer, it is harder to understand that all these functions will always be called in serial, and it requires many changes if we want to pass more data than a, b, and c to the last part of the code.

PEP 342 (see also the Python 2.5 release notes) and newdefgen will simplify the DeferredGenerator functionality. The history of newdefgen and the standard DeferredGenerator is here. The above code would become:

   @defgen
   def f(a, b, c):
       r = yield operationReturningDeferred().addErrback(g)
       print r, a, b, c
       r = yield anotherOperationReturningDeferred()
       print r, a, b, c
       r = yield lastOperationReturningDeferred()
       print r, a, b, c
       yield 6

However, PEP 342 is part of Python 2.5 and will not be released until August 2006. It will probably 2007 until it is included in the popular distributions and widely deployed, so we will stay with the current implementation until then.