One up to Python expert (1) - Decorators

I have been using Python for almost 4 years. I still remembered how amazed I was by the elegance of code indention, simplicity at the first sight, started from the official tutorial, then dive into it, daily scripts, then couple of side projects, but until today, I still hesitate to claim a Python expert. I would rather share this series of the journey to broaden my vision and deepen my insight on this beautiful language.

Introduction

Dr. David Mertz has a stunning demonstration in Charming Python column show this language feature, decorater. Here is the sample code to add spam, the favorite food for python, to arbitrary function:

def addspam(fn):
    def new(*args):
        "new method"
        print "spam spam spam"
        return fn(*args)
    return new

@addspam
def add(a,b):
    "add method"
     print a**2 + b**2

if __name__ == "__main__":
    add(3, 4)

The output of is:

spam spam spam
25

According to the Python language reference, the @ operation is defined as:

Decorator expressions are evaluated when the function is defined, in the scope that contains the function definition. The result must be a callable, which is invoked with the function object as the only argument. The returned value is bound to the function name instead of the function object. Multiple decorators are applied in nested fashion.

Therefore, our add is defined as:

add = addspam(add)

new is an anonymous function, the name is arbitrary since we would never call it directly. When add is invoked, addspam is evaluated and returns a callable object, new, which accepts the arguments, then is executed. As a well-behaviored decorator, new eventually calls the decoratee after spreading the word, “spam”.

Before we rush to more sophisticated application, let’s take a look at the flaws of the crystal ball. Yes, it is not crystal transparent, new blocks the signature of add:

>>> print add.__doc__
new method

We can copy the meta data by all means, but is there a smart way to avoid the boilerplate code? Yes, we can use Michele Simionato’s decorator library like this:

from decorator import decorator
@decorator
def addspam(fn, *args, **kw):
    "new method"
    print "spam spam spam", args
    return fn(*args)

@addspam
def add(a,b):
    "add method"
    print a**2 + b**2

if __name__ == "__main__":
    print add.__doc__
    add(3, 4)

That is quite dizzying, What is the on the earth under the hood?

Under the hood

First, let us inspect the vanilla version, each decorator would block the signature of the decoratee, illustrated by different colors.

Here is the code snippet of decorator.py:

def decorator(caller):
    def _decorator(func): # the real meat is here
        infodict = getinfo(func)
        argnames = infodict['argnames']
        assert not ('_call_' in argnames or '_func_' in argnames), (
            'You cannot use _call_ or _func_ as argument names!')
        src = "lambda %(signature)s: _call_(_func_, %(signature)s)" % infodict
        # import sys; print >> sys.stderr, src # for debugging purposes
        dec_func = eval(src, dict(_func_=func, _call_=caller))
        return update_wrapper(dec_func, func, infodict)
    return update_wrapper(_decorator, caller)

The first difference that caught my eye was the argument of _decorator, func, instead of *argv, does this matter?

Yes, that is the trick of the magic. decorator decorates the addspam which decoratesadd, are you still awake? So _add_is the argument for decorator’s anonymous function, i.e __decorator_.

There are two assistants for the magic: getinfo copy the signature of the function; update_wrapper signs the _decorator_with _caller’s signature. When add is invoked, decorator(addspam) is evaluated, which returns _decortor with addspam’s signature, in another word, decorator is transparent to addspam. _decorator is also the decorator of add, so _decorator(add) is executed:

1. Sanity check: make sure keyword is not used in the argument names
2. Cook the real meat: build addspam(add) in dec_func. Please check 3.6.2 String Formatting Operations for the syntax of mapping dictionary.
3. Seal the can with fun(i.e add)‘s signature

Here is the dynamic illustration:

Patterns

I would discuss this topic in detail later, you could take a look at official wiki.

Memorize

This pattern is well-documented in Wikipedia, and here is an effective but obtrusive implementation in DDJ¹, the corresponding python implementation is much more intuitive:

@memoize
def fib(n):
    print "%d is caculated" % n
    if  n < 2 :
        return 1;
    else:
    return fib(n-1) + fib(n-2)

Programming by Contract

Programming by Contract is an approach for software engineering. Microsoft Visual C++ introduced SAL annotations for precondition and postcondition. Decorators helps to separate the contract and logic²:

@precondition("tom > 0")
@precondition("jerry > 1")
@postcondition(lambda x: x > 1)
def foo(tom=1, jerry=2, rose=3, jack=4):
    print tom
    return jack

Precondition determines the requirement of the arguments, it is more convenient to use names of arguments for evaluation; postcondition specifies the return value, function object is more appropriate to refer the returned value. Here is the full implementation.

Aspect oriented programming

AOP is quite popular in Java community, there is corresponding Python project, PEAK for enterprise environment. For lightweight AOP developer, the decorator could do some help, such as the canonical fund transfer example:

@precondition("amount > 0")
@precondition("fromAccount.amount > amount")
def transfer(fromAccount, toAccount, amount):
    # TODO: add transaction here.
    fromAccount.withdraw(amount)
    toAccount.deposit(amount)

Conclusion

Decorator opens a door to override the default behavior of function. The add-on lets the magic shine and hide the mechanism behind the curtain.