Twisted Documentation: Deferreds are beautiful! (A Tutorial)

# Deferreds are beautiful! (A Tutorial)

## Introduction

Deferreds are quite possibly the single most confusing topic that a newcomer to Twisted has to deal with. I am going to forgo the normal talk about what deferreds are, what they aren't, and why they're used in Twisted. Instead, I'm going show you the logic behind what they do.

A deferred allows you to encapsulate the logic that you'd normally use to make a series of function calls after receiving a result into a single object. In the examples that follow, I'll first show you what's going to go on behind the scenes in the deferred chain, then show you the deferred API calls that set up that chain. All of these examples are runnable code, so feel free to play around with them.

## A simple example

First, a simple example so that we have something to talk about:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    #!/usr/bin/env python

    # Copyright (c) Twisted Matrix Laboratories.
    # See LICENSE for details.

    from twisted.internet import defer
    from twisted.python import failure, util

    """
    Here we have the simplest case, a single callback and a single errback.
    """

    num = 0

    def handleFailure(f):
        print "errback"
        print "we got an exception: %s" % (f.getTraceback(),)
        f.trap(RuntimeError)

    def handleResult(result):
        global num; num += 1
        print "callback %s" % (num,)
        print "\tgot result: %s" % (result,)
        return "yay! handleResult was successful!"

    def behindTheScenes(result):
        # equivalent to d.callback(result)

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = handleResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        if not isinstance(result, failure.Failure): # ---- callback
            pass
        else:                                       # ---- errback
            try:
                result = handleFailure(result)
            except:
                result = failure.Failure()

    def deferredExample():
        d = defer.Deferred()
        d.addCallback(handleResult)
        d.addErrback(handleFailure)

        d.callback("success")

    if __name__ == '__main__':
        behindTheScenes("success")
        print "\n-------------------------------------------------\n"
        global num; num = 0
        deferredExample()

Source listing - listings/deferred/deferred_ex.py

And the output: (since both methods in the example produce the same output, it will only be shown once.)

    callback 1
            got result: success

Here we have the simplest case. A deferred with a single callback and a single errback. Normally, a function would create a deferred and hand it back to you when you request an operation that needs to wait for an event for completion. The object you called then does `d.callback(result)` when the results are in.

The thing to notice is that there is only one result that is passed from method to method, and that the result returned from a method is the argument to the next method in the chain. In case of an exception, result is set to an instance of `Failure` that describes the exception.

## Errbacks

### Failure in requested operation

Things don't always go as planned, and sometimes the function that returned the deferred needs to alert the callback chain that an error has occurred.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    #!/usr/bin/env python

    # Copyright (c) Twisted Matrix Laboratories.
    # See LICENSE for details.

    from twisted.internet import defer
    from twisted.python import failure, util

    """
    This example is analogous to a function calling .errback(failure)
    """

    class Counter(object):
        num = 0

    def handleFailure(f):
        print "errback"
        print "we got an exception: %s" % (f.getTraceback(),)
        f.trap(RuntimeError)

    def handleResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        return "yay! handleResult was successful!"

    def failAtHandlingResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        print "\tabout to raise exception"
        raise RuntimeError, "whoops! we encountered an error"

    def behindTheScenes(result):
        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = handleResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        if not isinstance(result, failure.Failure): # ---- callback
            pass
        else:                                       # ---- errback
            try:
                result = handleFailure(result)
            except:
                result = failure.Failure()

    def deferredExample(result):
        d = defer.Deferred()
        d.addCallback(handleResult)
        d.addCallback(failAtHandlingResult)
        d.addErrback(handleFailure)

        d.errback(result)

    if __name__ == '__main__':
        result = None
        try:
            raise RuntimeError, "*doh*! failure!"
        except:
            result = failure.Failure()
        behindTheScenes(result)
        print "\n-------------------------------------------------\n"
        Counter.num = 0
        deferredExample(result)

Source listing - listings/deferred/deferred_ex1a.py

    errback
    we got an exception: Traceback (most recent call last):
    --- exception caught here ---
      File "deferred_ex1a.py", line 73, in ?
        raise RuntimeError, "*doh*! failure!"
    exceptions.RuntimeError: *doh*! failure!

The important thing to note (as it will come up again in later examples) is that the callback isn't touched, the failure goes right to the errback. Also note that the errback trap()s the expected exception type. If you don't trap the exception, an error will be logged when the deferred is garbage-collected.

### Exceptions raised in callbacks

Now let's see what happens when our callback raises an exception

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    #!/usr/bin/env python

    # Copyright (c) Twisted Matrix Laboratories.
    # See LICENSE for details.

    from twisted.internet import defer
    from twisted.python import failure, util

    """
    Here we have a slightly more involved case. The deferred is called back with a
    result. the first callback returns a value, the second callback, however
    raises an exception, which is handled by the errback.
    """

    class Counter(object):
        num = 0

    def handleFailure(f):
        print "errback"
        print "we got an exception: %s" % (f.getTraceback(),)
        f.trap(RuntimeError)

    def handleResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        return "yay! handleResult was successful!"

    def failAtHandlingResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        print "\tabout to raise exception"
        raise RuntimeError, "whoops! we encountered an error"

    def behindTheScenes(result):
        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = handleResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = failAtHandlingResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        if not isinstance(result, failure.Failure): # ---- callback
            pass
        else:                                       # ---- errback
            try:
                result = handleFailure(result)
            except:
                result = failure.Failure()

    def deferredExample():
        d = defer.Deferred()
        d.addCallback(handleResult)
        d.addCallback(failAtHandlingResult)
        d.addErrback(handleFailure)

        d.callback("success")

    if __name__ == '__main__':
        behindTheScenes("success")
        print "\n-------------------------------------------------\n"
        Counter.num = 0
        deferredExample()

Source listing - listings/deferred/deferred_ex1b.py

And the output: (note, tracebacks will be edited slightly to conserve space)

    callback 1
            got result: success
    callback 2
            got result: yay! handleResult was successful!
            about to raise exception
    errback
    we got an exception: Traceback (most recent call last):
    --- <exception caught here> ---
      File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line
    326, in _runCallbacks
        self.result = callback(self.result, *args, **kw)
      File "./deferred_ex1.py", line 32, in failAtHandlingResult
        raise RuntimeError, "whoops! we encountered an error"
    exceptions.RuntimeError: whoops! we encountered an error

If your callback raises an exception, the next method to be called will be the next errback in your chain.

### Exceptions will only be handled by errbacks

If a callback raises an exception the next method to be called will be next errback in the chain. If the chain is started off with a failure, the first method to be called will be the first errback.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    #!/usr/bin/env python
    # Copyright (c) Twisted Matrix Laboratories.
    # See LICENSE for details.

    from twisted.internet import defer
    from twisted.python import failure

    """
    This example shows an important concept that many deferred newbies have trouble
    understanding.

    When an error occurs in a callback, the first errback after the error occurs
    will be the next method called. (In the next example we'll see what happens in
    the 'chain' after an errback.)
    """

    class Counter(object):
        num = 0

    def handleFailure(f):
        print "errback"
        print "we got an exception: %s" % (f.getTraceback(),)
        f.trap(RuntimeError)

    def handleResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        return "yay! handleResult was successful!"

    def failAtHandlingResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        print "\tabout to raise exception"
        raise RuntimeError, "whoops! we encountered an error"

    def behindTheScenes(result):
        # equivalent to d.callback(result)

        # now, let's make the error happen in the first callback

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = failAtHandlingResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        # note: this callback will be skipped because
        # result is a failure

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = handleResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        if not isinstance(result, failure.Failure): # ---- callback
            pass
        else:                                       # ---- errback
            try:
                result = handleFailure(result)
            except:
                result = failure.Failure()

    def deferredExample():
        d = defer.Deferred()
        d.addCallback(failAtHandlingResult)
        d.addCallback(handleResult)
        d.addErrback(handleFailure)

        d.callback("success")

    if __name__ == '__main__':
        behindTheScenes("success")
        print "\n-------------------------------------------------\n"
        Counter.num = 0
        deferredExample()

Source listing - listings/deferred/deferred_ex2.py

    callback 1
    	got result: success
    	about to raise exception
    errback
    we got an exception: Traceback (most recent call last):
      File "deferred_ex2.py", line 90, in <module>
        deferredExample()
      File "deferred_ex2.py", line 83, in deferredExample
        d.callback("success")
      File "twisted/internet/defer.py", line 361, in callback
        self._startRunCallbacks(result)
      File "twisted/internet/defer.py", line 455, in _startRunCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "twisted/internet/defer.py", line 542, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "deferred_ex2.py", line 37, in failAtHandlingResult
        raise RuntimeError, "whoops! we encountered an error"
    exceptions.RuntimeError: whoops! we encountered an error

You can see that our second callback, handleResult was not called because failAtHandlingResult raised an exception

### Handling an exception and continuing on

In this example, we see an errback handle an exception raised in the preceeding callback. Take note that it could just as easily been an exception from any other preceeding method. You'll see that after the exception is handled in the errback (i.e. the errback does not return a failure or raise an exception) the chain continues on with the next callback.

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
     47
     48
     49
     50
     51
     52
     53
     54
     55
     56
     57
     58
     59
     60
     61
     62
     63
     64
     65
     66
     67
     68
     69
     70
     71
     72
     73
     74
     75
     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    #!/usr/bin/env python

    # Copyright (c) Twisted Matrix Laboratories.
    # See LICENSE for details.

    from twisted.internet import defer
    from twisted.python import failure, util

    """
    Now we see how an errback can handle errors. if an errback
    does not raise an exception, the next callback in the chain
    will be called.
    """

    class Counter(object):
        num = 0

    def handleFailure(f):
        print "errback"
        print "we got an exception: %s" % (f.getTraceback(),)
        f.trap(RuntimeError)
        return "okay, continue on"

    def handleResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        return "yay! handleResult was successful!"

    def failAtHandlingResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        print "\tabout to raise exception"
        raise RuntimeError, "whoops! we encountered an error"

    def callbackAfterErrback(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)

    def behindTheScenes(result):
        # equivalent to d.callback(result)

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = handleResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = failAtHandlingResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        if not isinstance(result, failure.Failure): # ---- callback
            pass
        else:                                       # ---- errback
            try:
                result = handleFailure(result)
            except:
                result = failure.Failure()

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = callbackAfterErrback(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

    def deferredExample():
        d = defer.Deferred()
        d.addCallback(handleResult)
        d.addCallback(failAtHandlingResult)
        d.addErrback(handleFailure)
        d.addCallback(callbackAfterErrback)

        d.callback("success")

    if __name__ == '__main__':
        behindTheScenes("success")
        print "\n-------------------------------------------------\n"
        Counter.num = 0
        deferredExample()

Source listing - listings/deferred/deferred_ex3.py

    callback 1
            got result: success
    callback 2
            got result: yay! handleResult was successful!
            about to raise exception
    errback
    we got an exception: Traceback (most recent call last):
      File "./deferred_ex3.py", line 97, in <module>
        deferredExample()
      File "./deferred_ex3.py", line 90, in deferredExample
        d.callback("success")
      File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line 243, in callback
        self._startRunCallbacks(result)
      File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line 312, in _startRunCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line 328, in _runCallbacks
        self.result = callback(self.result, *args, **kw)
      File "./deferred_ex3.py", line 34, in failAtHandlingResult
        raise RuntimeError, "whoops! we encountered an error"
    exceptions.RuntimeError: whoops! we encountered an error

    callback 3
            got result: okay, continue on

## addBoth: the deferred version of finally

Now we see how deferreds do finally, with .addBoth. The callback that gets added as addBoth will be called if the result is a failure or non-failure. We'll also see in this example, that our doThisNoMatterWhat() method follows a common idiom in deferred callbacks by acting as a passthru, returning the value that it received to allow processing the chain to continue, but appearing transparent in terms of the result.

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
     47
     48
     49
     50
     51
     52
     53
     54
     55
     56
     57
     58
     59
     60
     61
     62
     63
     64
     65
     66
     67
     68
     69
     70
     71
     72
     73
     74
     75
     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    101
    102
    103
    104
    #!/usr/bin/env python

    # Copyright (c) Twisted Matrix Laboratories.
    # See LICENSE for details.

    from twisted.internet import defer
    from twisted.python import failure, util

    """
    Now we'll see what happens when you use 'addBoth'.
    """

    class Counter(object):
        num = 0

    def handleFailure(f):
        print "errback"
        print "we got an exception: %s" % (f.getTraceback(),)
        f.trap(RuntimeError)
        return "okay, continue on"

    def handleResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        return "yay! handleResult was successful!"

    def failAtHandlingResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        print "\tabout to raise exception"
        raise RuntimeError, "whoops! we encountered an error"

    def doThisNoMatterWhat(arg):
        Counter.num += 1
        print "both %s" % (Counter.num,)
        print "\tgot argument %r" % (arg,)
        print "\tdoing something very important"
        # we pass the argument we received to the next phase here
        return arg

    def behindTheScenes(result):
        # equivalent to d.callback(result)

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = handleResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = failAtHandlingResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        # ---- this is equivalent to addBoth(doThisNoMatterWhat)

        if not isinstance(result, failure.Failure):
            try:
                result = doThisNoMatterWhat(result)
            except:
                result = failure.Failure()
        else:
            try:
                result = doThisNoMatterWhat(result)
            except:
                result = failure.Failure()

        if not isinstance(result, failure.Failure): # ---- callback
            pass
        else:                                       # ---- errback
            try:
                result = handleFailure(result)
            except:
                result = failure.Failure()

    def deferredExample():
        d = defer.Deferred()
        d.addCallback(handleResult)
        d.addCallback(failAtHandlingResult)
        d.addBoth(doThisNoMatterWhat)
        d.addErrback(handleFailure)

        d.callback("success")

    if __name__ == '__main__':
        behindTheScenes("success")
        print "\n-------------------------------------------------\n"
        Counter.num = 0
        deferredExample()

Source listing - listings/deferred/deferred_ex4.py

    callback 1
            got result: success
    callback 2
            got result: yay! handleResult was successful!
            about to raise exception
    both 3
            got argument <twisted.python.failure.Failure exceptions.RuntimeError>
            doing something very important
    errback
    we got an exception: Traceback (most recent call last):
    --- <exception caught here> ---
      File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line
    326, in _runCallbacks
        self.result = callback(self.result, *args, **kw)
      File "./deferred_ex4.py", line 32, in failAtHandlingResult
        raise RuntimeError, "whoops! we encountered an error"
    exceptions.RuntimeError: whoops! we encountered an error

You can see that the errback is called, (and consequently, the failure is trapped). This is because doThisNoMatterWhat method returned the value it received, a failure.

## addCallbacks: decision making based on previous success or failure

As we've been seeing in the examples, the callback is a pair of callback/errback. Using addCallback or addErrback is actually a special case where one of the pair is a pass statement. If you want to make a decision based on whether or not the previous result in the chain was a failure or not (which is very rare, but included here for completeness), you use addCallbacks. Note that this is not the same thing as an addCallback followed by an addErrback.

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
     47
     48
     49
     50
     51
     52
     53
     54
     55
     56
     57
     58
     59
     60
     61
     62
     63
     64
     65
     66
     67
     68
     69
     70
     71
     72
     73
     74
     75
     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    #!/usr/bin/env python

    # Copyright (c) Twisted Matrix Laboratories.
    # See LICENSE for details.

    from twisted.internet import defer
    from twisted.python import failure, util

    """
    Now comes the more nuanced addCallbacks, which allows us to make a
    yes/no (branching) decision based on whether the result at a given point is
    a failure or not.
    """

    class Counter(object):
        num = 0

    def handleFailure(f):
        print "errback"
        print "we got an exception: %s" % (f.getTraceback(),)
        f.trap(RuntimeError)
        return "okay, continue on"

    def handleResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        return "yay! handleResult was successful!"

    def failAtHandlingResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        print "\tabout to raise exception"
        raise RuntimeError, "whoops! we encountered an error"

    def yesDecision(result):
        Counter.num += 1
        print "yes decision %s" % (Counter.num,)
        print "\twasn't a failure, so we can plow ahead"
        return "go ahead!"

    def noDecision(result):
        Counter.num += 1
        result.trap(RuntimeError)
        print "no decision %s" % (Counter.num,)
        print "\t*doh*! a failure! quick! damage control!"
        return "damage control successful!"

    def behindTheScenes(result):

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = failAtHandlingResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        # this is equivalent to addCallbacks(yesDecision, noDecision)

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = yesDecision(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            try:
                result = noDecision(result)
            except:
                result = failure.Failure()

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = handleResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        # this is equivalent to addCallbacks(yesDecision, noDecision)

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = yesDecision(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            try:
                result = noDecision(result)
            except:
                result = failure.Failure()

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = handleResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        if not isinstance(result, failure.Failure): # ---- callback
            pass
        else:                                       # ---- errback
            try:
                result = handleFailure(result)
            except:
                result = failure.Failure()

    def deferredExample():
        d = defer.Deferred()
        d.addCallback(failAtHandlingResult)
        d.addCallbacks(yesDecision, noDecision) # noDecision will be called
        d.addCallback(handleResult) # - A -
        d.addCallbacks(yesDecision, noDecision) # yesDecision will be called
        d.addCallback(handleResult)
        d.addErrback(handleFailure)

        d.callback("success")

    if __name__ == '__main__':
        behindTheScenes("success")
        print "\n-------------------------------------------------\n"
        Counter.num = 0
        deferredExample()

Source listing - listings/deferred/deferred_ex5.py

    callback 1
            got result: success
            about to raise exception
    no decision 2
            *doh*! a failure! quick! damage control!
    callback 3
            got result: damage control successful!
    yes decision 4
            wasn't a failure, so we can plow ahead
    callback 5
            got result: go ahead!

Notice that our errback is never called. The noDecision method returns a non-failure so processing continues with the next callback. If we wanted to skip the callback at "- A -" because of the error, but do some kind of processing in response to the error, we would have used a passthru, and returned the failure we received, as we see in this next example:

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
     47
     48
     49
     50
     51
     52
     53
     54
     55
     56
     57
     58
     59
     60
     61
     62
     63
     64
     65
     66
     67
     68
     69
     70
     71
     72
     73
     74
     75
     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    #!/usr/bin/env python

    # Copyright (c) Twisted Matrix Laboratories.
    # See LICENSE for details.

    from twisted.internet import defer
    from twisted.python import failure, util

    """
    Now comes the more nuanced addCallbacks, which allows us to make a
    yes/no (branching) decision based on whether the result at a given point is
    a failure or not.

    here, we return the failure from noDecisionPassthru, the errback argument to
    the first addCallbacks method invocation, and see what happens.
    """

    class Counter(object):
        num = 0

    def handleFailure(f):
        print "errback"
        print "we got an exception: %s" % (f.getTraceback(),)
        f.trap(RuntimeError)
        return "okay, continue on"

    def handleResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        return "yay! handleResult was successful!"

    def failAtHandlingResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        print "\tabout to raise exception"
        raise RuntimeError, "whoops! we encountered an error"

    def yesDecision(result):
        Counter.num += 1
        print "yes decision %s" % (Counter.num,)
        print "\twasn't a failure, so we can plow ahead"
        return "go ahead!"

    def noDecision(result):
        Counter.num += 1
        result.trap(RuntimeError)
        print "no decision %s" % (Counter.num,)
        print "\t*doh*! a failure! quick! damage control!"
        return "damage control successful!"

    def noDecisionPassthru(result):
        Counter.num += 1
        print "no decision %s" % (Counter.num,)
        print "\t*doh*! a failure! don't know what to do, returning failure!"
        return result

    def behindTheScenes(result):

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = failAtHandlingResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        # this is equivalent to addCallbacks(yesDecision, noDecision)

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = yesDecision(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            try:
                result = noDecisionPassthru(result)
            except:
                result = failure.Failure()

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = handleResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        # this is equivalent to addCallbacks(yesDecision, noDecision)

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = yesDecision(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            try:
                result = noDecision(result)
            except:
                result = failure.Failure()

        if not isinstance(result, failure.Failure): # ---- callback
            try:
                result = handleResult(result)
            except:
                result = failure.Failure()
        else:                                       # ---- errback
            pass

        if not isinstance(result, failure.Failure): # ---- callback
            pass
        else:                                       # ---- errback
            try:
                result = handleFailure(result)
            except:
                result = failure.Failure()

    def deferredExample():
        d = defer.Deferred()
        d.addCallback(failAtHandlingResult)

        # noDecisionPassthru will be called
        d.addCallbacks(yesDecision, noDecisionPassthru)
        d.addCallback(handleResult) # - A -

        # noDecision will be called
        d.addCallbacks(yesDecision, noDecision)
        d.addCallback(handleResult) # - B -
        d.addErrback(handleFailure)

        d.callback("success")

    if __name__ == '__main__':
        behindTheScenes("success")
        print "\n-------------------------------------------------\n"
        Counter.num = 0
        deferredExample()

Source listing - listings/deferred/deferred_ex6.py

    callback 1
            got result: success
            about to raise exception
    no decision 2
            *doh*! a failure! don't know what to do, returning failure!
    no decision 3
            *doh*! a failure! quick! damage control!
    callback 4
            got result: damage control successful!

Two things to note here. First, "- A -" was skipped, like we wanted it to, and the second thing is that after "- A -", noDecision is called, because it is the next errback that exists in the chain. It returns a non-failure, so processing continues with the next callback at "- B -", and the errback at the end of the chain is never called

## Hints, tips, common mistakes, and miscellaney

### The deferred callback chain is stateful

A deferred that has been called back will call its addCallback and addErrback methods as appropriate in the order they are added, when they are added. So we see in the following example, deferredExample1 and deferredExample2 are equivalent. The first sets up the processing chain beforehand and then executes it, the other executes the chain as it is being constructed. This is because deferreds are stateful.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    #!/usr/bin/env python

    # Copyright (c) Twisted Matrix Laboratories.
    # See LICENSE for details.

    from twisted.internet import defer
    from twisted.python import failure, util

    """
    The deferred callback chain is stateful, and can be executed before
    or after all callbacks have been added to the chain
    """

    class Counter(object):
        num = 0

    def handleFailure(f):
        print "errback"
        print "we got an exception: %s" % (f.getTraceback(),)
        f.trap(RuntimeError)

    def handleResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        return "yay! handleResult was successful!"

    def failAtHandlingResult(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        print "\tabout to raise exception"
        raise RuntimeError, "whoops! we encountered an error"

    def deferredExample1():
        # this is another common idiom, since all add* methods
        # return the deferred instance, you can just chain your
        # calls to addCallback and addErrback

        d = defer.Deferred().addCallback(failAtHandlingResult
                           ).addCallback(handleResult
                           ).addErrback(handleFailure)

        d.callback("success")

    def deferredExample2():
        d = defer.Deferred()

        d.callback("success")

        d.addCallback(failAtHandlingResult)
        d.addCallback(handleResult)
        d.addErrback(handleFailure)

    if __name__ == '__main__':
        deferredExample1()
        print "\n-------------------------------------------------\n"
        Counter.num = 0
        deferredExample2()

Source listing - listings/deferred/deferred_ex7.py

    callback 1
            got result: success
            about to raise exception
    errback
    we got an exception: Traceback (most recent call last):
    --- <exception caught here> ---
      File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line
    326, in _runCallbacks
        self.result = callback(self.result, *args, **kw)
      File "./deferred_ex7.py", line 35, in failAtHandlingResult
        raise RuntimeError, "whoops! we encountered an error"
    exceptions.RuntimeError: whoops! we encountered an error

    -------------------------------------------------

    callback 1
            got result: success
            about to raise exception
    errback
    we got an exception: Traceback (most recent call last):
    --- <exception caught here> ---
      File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line
    326, in _runCallbacks
        self.result = callback(self.result, *args, **kw)
      File "./deferred_ex7.py", line 35, in failAtHandlingResult
        raise RuntimeError, "whoops! we encountered an error"
    exceptions.RuntimeError: whoops! we encountered an error

This example also shows you the common idiom of chaining calls to addCallback and addErrback.

### Don't call .callback() on deferreds you didn't create!

It is an error to reinvoke deferreds callback or errback method, therefore if you didn't create a deferred, do not under any circumstances call its callback or errback. doing so will raise an exception

### Callbacks can return deferreds

If you need to call a method that returns a deferred within your callback chain, just return that deferred, and the result of the secondary deferred's processing chain will become the result that gets passed to the next callback of the primary deferreds processing chain

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    #!/usr/bin/env python

    # Copyright (c) Twisted Matrix Laboratories.
    # See LICENSE for details.

    from twisted.internet import defer
    from twisted.python import failure, util

    class Counter(object):
        num = 0
        let = 'a'

        def incrLet(cls):
            cls.let = chr(ord(cls.let) + 1)
        incrLet = classmethod(incrLet)

    def handleFailure(f):
        print "errback"
        print "we got an exception: %s" % (f.getTraceback(),)
        return f

    def subCb_B(result):
        print "sub-callback %s" % (Counter.let,)
        Counter.incrLet()
        s = " beautiful!"
        print "\tadding %r to result" % (s,)
        result += s
        return result

    def subCb_A(result):
        print "sub-callback %s" % (Counter.let,)
        Counter.incrLet()
        s = " are "
        print "\tadding %r to result" % (s,)
        result += s
        return result

    def mainCb_1(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)
        result += " Deferreds "

        d = defer.Deferred().addCallback(subCb_A
                           ).addCallback(subCb_B)
        d.callback(result)
        return d

    def mainCb_2(result):
        Counter.num += 1
        print "callback %s" % (Counter.num,)
        print "\tgot result: %s" % (result,)

    def deferredExample():
        d = defer.Deferred().addCallback(mainCb_1
                           ).addCallback(mainCb_2)

        d.callback("I hope you'll agree: ")

    if __name__ == '__main__':
        deferredExample()

Source listing - listings/deferred/deferred_ex8.py

    callback 1
            got result: I hope you'll agree:
    sub-callback a
            adding ' are ' to result
    sub-callback b
            adding ' beautiful!' to result
    callback 2
            got result: I hope you'll agree:  Deferreds  are  beautiful!

## Conclusion

Deferreds can be confusing, but only because they're so elegant and simple. There is a lot of logical power that can expressed with a deferred's processing chain, and once you see what's going on behind the curtain, it's a lot easier to understand how to make use of what deferreds have to offer.

Index

Version: 11.1.0