Decorators
A decorator is a design pattern in which a class or function alters or adds to the functionality of another class or function without using inheritance, or directly modifying the source code. In Python, decorators are, in simplest terms, functions (or any callable objects) that take as input a set of optional arguments and a function or class, and return a function or class. They can be used to implement the decorator design pattern, or for other purposes. Class decorators are new in Python 2.6.
By the way, before you read further, if you are unfamiliar with Python's closures, make sure to familiarize yourself with them by looking at the closures appendix at the end of this article, as decorators can be hard to understand without a proper understanding of closures in Python.
Decorators are applied to a function or class using the @ symbol in Python. Let's use a simple decorator to log function calls as our first example. Here the decorator takes the time format as an argument and prints a log statement prior to and after every execution of the decorated function with the execution time. This can come in handy when you are comparing the efficiency of two different implementations of the same algorithm, or two separate algorithms, for example.
def logged(time_format):
def decorator(func):
def decorated_func(*args, **kwargs):
print "- Running '%s' on %s " % (
func.__name__,
time.strftime(time_format)
)
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print "- Finished '%s', execution time = %0.3fs " % (
func.__name__,
end_time - start_time
)
return result
decorated_func.__name__ = func.__name__
return decorated_func
return decorator
Let's look at a use example. Here functions add1 and add2 are decorated using logged and a sample output is given. Notice that the time format argument is stored in the closure of the returned decorated functions. This is why an understanding of closures is important in understanding decorators in Python. Also notice how the returned function's name is replaced with the original function's name, in case it is used later on, to prevent confusion. Python does not do this by default.
@logged("%b %d %Y - %H:%M:%S")
def add1(x, y):
time.sleep(1)
return x + y
@logged("%b %d %Y - %H:%M:%S")
def add2(x, y):
time.sleep(2)
return x + y
print add1(1, 2)
print add2(1, 2)
# Output:
- Running 'add1' on Jul 24 2013 - 13:40:47
- Finished 'add1', execution time = 1.001s
3
- Running 'add2' on Jul 24 2013 - 13:40:48
- Finished 'add2', execution time = 2.001s
3
If you pay careful attention, you might notice that while we take extra care to make sure the returned function has the right __name__, we do not do so for __doc__, or __module__. So if, in our example, the add function had a doc string, it would be lost. How do we handle this? We could handle it similar to how we handled __name__ but doing so for every decorator becomes tedious. This is why the functools module provides a decorator named wraps which handles exactly this scenario. It might be a bit confusing to see a decorator used inside another, but when you think of decorators as simply functions that take in a function as parameter and return a function, everything makes perfect sense. The wraps decorator is used in our next examples instead of manually fixing __name__ and other such attributes.
Next example is a bit more complicated. Let's write a decorator that caches the result of a function call for a given number of seconds. The code relies on the arguments passed to the function to be hashable objects because we use a tuple with the args arguments as the first entry, and a frozen set of the items in the keyword arguments kwargs as the second entry as the cache key. Each function will have a unique cache dict generated for it which is stored in the function's closure.
import time
from functools import wraps
def cached(timeout, logged=False):
"""Decorator to cache the result of a function call.
Cache expires after timeout seconds.
"""
def decorator(func):
if logged:
print "-- Initializing cache for", func.__name__
cache = {}
@wraps(func)
def decorated_function(*args, **kwargs):
if logged:
print "-- Called function", func.__name__
key = (args, frozenset(kwargs.items()))
result = None
if key in cache:
if logged:
print "-- Cache hit for", func.__name__, key
(cache_hit, expiry) = cache[key]
if time.time() - expiry < timeout:
result = cache_hit
elif logged:
print "-- Cache expired for", func.__name__, key
elif logged:
print "-- Cache miss for", func.__name__, key
# No cache hit, or expired
if result is None:
result = func(*args, **kwargs)
cache[key] = (result, time.time())
return result
return decorated_function
return decorator
And here's how it's used. We apply the decorator to a very naive (and inefficient) Fibonacci number calculator. The cache decorator will effectively apply the memoize pattern to the code. Notice how the closure of fib contains the cache dict, a reference to the original fib function, the value of the logged argument, and finally the value of the timeout argument. dump_closure is defined at the end of the article, under the closures section.
>>> @cached(10, True)
... def fib(n):
... """Returns the n'th Fibonacci number."""
... if n == 0 or n == 1:
... return 1
... return fib(n - 1) + fib(n - 2)
...
-- Initializing cache for fib
>>> dump_closure(fib)
1. Dumping function closure for fib:
-- cell 0 = {}
-- cell 1 = <function fib at 0x10eae7500>
-- cell 2 = True
-- cell 3 = 10
>>>
>>> print "Testing - F(4) = %d" % fib(4)
-- Called function fib
-- Cache miss for fib ((4,), frozenset([]))
-- Called function fib
-- Cache miss for fib ((3,), frozenset([]))
-- Called function fib
-- Cache miss for fib ((2,), frozenset([]))
-- Called function fib
-- Cache miss for fib ((1,), frozenset([]))
-- Called function fib
-- Cache miss for fib ((0,), frozenset([]))
-- Called function fib
-- Cache hit for fib ((1,), frozenset([]))
-- Called function fib
-- Cache hit for fib ((2,), frozenset([]))
Testing - F(4) = 5
Class Decorators
In the previous section we looked at function decorators and some non-trivial examples of their uses. Next, let's look at class decorators. Here, the decorator takes as input a class (an object of type type in Python), and returns a modified class.
First example is a simple mathematical example. Given a partially ordered set (also called a poset) , we define to be the dual of if and only if . In other words, the ordering is reversed. How can we implement this in Python? Suppose a class defines an ordering by implementing the __lt__, __le__, etc. set of methods. Then we can write a class decorator that replaces each of these functions with its dual.
def make_dual(relation):
@wraps(relation, ['__name__', '__doc__'])
def dual(x, y):
return relation(y, x)
return dual
def dual_ordering(cls):
"""Class decorator that reverses all the orderings"""
for func in ['__lt__', '__gt__', '__ge__', '__le__']:
if hasattr(cls, func):
setattr(cls, func, make_dual(getattr(cls, func)))
return cls
Here's how we can apply that to str, to create a new class called rstr in which opposite lexicographic ordering is used.
@dual_ordering
class rstr(str):
pass
x = rstr("1")
y = rstr("2")
print x < y
print x <= y
print x > y
print x >= y
# Output:
False
False
True
True
Let's look at another more complicated example. Suppose we want the logged decorator from the previous section to be applied to all the methods in a class. One way to do this would be to go through the code and add the decorator to each method. Another would be write a class decorator that automates this process. Before I do this though, I took the logged decorator from the previous section and improved on it a bit. Firstly, it uses the wraps decorator from functools now instead of manually fixing the __name__. Secondly, a _logged_decorator attribute is added to the returned function which is set to True, and is used to make sure the function does not get double-decorated, which comes in handy when we apply the decorator to classes which might inherit methods from another class. Finally, a name_prefix argument is added to allow for customization of the printed log message.
def logged(time_format, name_prefix=""):
def decorator(func):
if hasattr(func, '_logged_decorator') and func._logged_decorator:
return func
@wraps(func)
def decorated_func(*args, **kwargs):
start_time = time.time()
print "- Running '%s' on %s " % (
name_prefix + func.__name__,
time.strftime(time_format)
)
result = func(*args, **kwargs)
end_time = time.time()
print "- Finished '%s', execution time = %0.3fs " % (
name_prefix + func.__name__,
end_time - start_time
)
return result
decorated_func._logged_decorator = True
return decorated_func
return decorator
Now we are ready to write the class decorator.
def log_method_calls(time_format):
def decorator(cls):
for o in dir(cls):
if o.startswith('__'):
continue
a = getattr(cls, o)
if hasattr(a, '__call__'):
decorated_a = logged(time_format, cls.__name__ + ".")(a)
setattr(cls, o, decorated_a)
return cls
return decorator
And here's how it would be used. Notice how inheritance and overridden methods are handled here.
@log_method_calls("%b %d %Y - %H:%M:%S")
class A(object):
def test1(self):
print "test1"
@log_method_calls("%b %d %Y - %H:%M:%S")
class B(A):
def test1(self):
super(B, self).test1()
print "child test1"
def test2(self):
print "test2"
b = B()
b.test1()
b.test2()
# Output:
- Running 'B.test1' on Jul 24 2013 - 14:15:03
- Running 'A.test1' on Jul 24 2013 - 14:15:03
test1
- Finished 'A.test1', execution time = 0.000s
child test1
- Finished 'B.test1', execution time = 1.001s
- Running 'B.test2' on Jul 24 2013 - 14:15:04
test2
- Finished 'B.test2', execution time = 2.001s
Our first example of class decorators was to reverse the ordering methods of a class. A somewhat similar decorator, though arguably a lot more useful, that can come in handy would be one that, given only one of __lt__, __le__, __gt__, or __ge__ and __eq__, would implement the others to make a total ordering for the class. This is precisely what the functools.total_ordering decorator does. You can read up on the documentation here.
A Few Examples From Flask
Let's look at some other interesting uses of decorators from Flask.
Suppose that you want certain functions to output a warning message if they are called under certain circumstances, but only in debug mode. Instead of manually adding the code to the beginning of every function, you can write a decorator to do this. This is precisely what the following decorator taken from Flask's app.py file does.
def setupmethod(f):
"""Wraps a method so that it performs a check in debug mode if the
first request was already handled.
"""
def wrapper_func(self, *args, **kwargs):
if self.debug and self._got_first_request:
raise AssertionError('A setup function was called after the '
'first request was handled. This usually indicates a bug '
'in the application where a module was not imported '
'and decorators or other functionality was called too late.\n'
'To fix this make sure to import all your view modules, '
'database models and everything related at a central place '
'before the application starts serving requests.')
return f(self, *args, **kwargs)
return update_wrapper(wrapper_func, f)
A more interesting example is Flask's route decorator, which is defined in the class Flask. Note how a decorator can be a method in a class, and as such has self as the first parameter. The full source is in the app.py file. Notice that the decorator simply registers the decorated function as a URL handler by calling the add_url_rule function.
def route(self, rule, **options):
"""A decorator that is used to register a view function for a
given URL rule. This does the same thing as :meth:`add_url_rule`
but is intended for decorator usage::
@app.route('/')
def index():
return 'Hello World'
For more information refer to :ref:`url-route-registrations`.
:param rule: the URL rule as string
:param endpoint: the endpoint for the registered URL rule. Flask
itself assumes the name of the view function as
endpoint
:param options: the options to be forwarded to the underlying
:class:`~werkzeug.routing.Rule` object. A change
to Werkzeug is handling of method options. methods
is a list of methods this rule should be limited
to (`GET`, `POST` etc.). By default a rule
just listens for `GET` (and implicitly `HEAD`).
Starting with Flask 0.6, `OPTIONS` is implicitly
added and handled by the standard request handling.
"""
def decorator(f):
endpoint = options.pop('endpoint', None)
self.add_url_rule(rule, endpoint, f, **options)
return f
return decorator
Further Reading
The official Python Wiki has lots of more information on decorators if you're looking for further reading.
There is also David Beazley's excellent video on metaprogramming in Python 3.
Appendix: Closures
A function closure is a combination of a function and a set of references to the variables in the scope in which the function was defined. The latter is sometimes referred to as a referencing environment. This allows the function to be executed outside of where it is defined. In Python, the referencing environment is stored as a tuple of cells. You can access them by using the func_closure or __closure___ attributes. (In Python 3 only __closure__.) It is important thing to keep in mind is that the references are just references, and not deep copies of the objects. Of course this does not matter if the objects are immutable, but for mutable objects (say lists) it matters. An example further down illustrates this. Note that functions also have __globals__ which keeps the global referencing environment for where the function was defined.
Let's look at a simple example:
>>> def return_func_that_prints_s(s):
... def f():
... print s
... return f
...
>>> g = return_func_that_prints_s("Hello")
>>> h = return_func_that_prints_s("World")
>>> g()
Hello
>>> h()
World
>>> g is h
False
>>> h.__closure__
(<cell at 0x10d172398: str object at 0x10d170840>,)
>>> print [str(c.cell_contents) for c in g.__closure__]
['Hello']
>>> print [str(c.cell_contents) for c in h.__closure__]
['World']
And a slightly more complicated example. Make sure you understand why the code runs the way it does.
>>> def return_func_that_prints_list(z):
... def f():
... print z
... return f
...
>>> z = [1, 2]
>>> g = return_func_that_prints_list(z)
>>> g()
[1, 2]
>>> z.append(3)
>>> g()
[1, 2, 3]
>>> z = [1]
>>> g()
[1, 2, 3]
Finally, the dump_closure method that was used in this article is given here.
def dump_closure(f):
if hasattr(f, "__closure__") and f.__closure__ is not None:
print "- Dumping function closure for %s:" % f.__name__
for i, c in enumerate(f.__closure__):
print "-- cell %d = %s" % (i, c.cell_contents)
else:
print " - %s has no closure!" % f.__name__
Comments