Generators, decorators, context managers
Zbigniew Jędrzejewski-Szmek
George Mason University
Objectives
- show the use of generators, decorators, and context managers
- explain the mechanism behind the magic
- show how to write your own
Why?
- nicer (minimalistic, declarative, elegant)
- separation of concerns
- less bugs, easier refactoring
Part I: generators¶
In [3]:
def countdown(n):
while n > 0:
yield n
n -= 1
In [4]:
counter = countdown(3)
In [5]:
next(counter)
Out[5]:
In [6]:
next(counter)
Out[6]:
In [7]:
next(counter)
Out[7]:
In [8]:
next(counter)
In [74]:
import networkx as nx
G=nx.DiGraph()
G.add_edge("generator function", "generator", label='call')
G.add_edge("generator", "item", label='next()')
pos = {"generator function":(0,3), "generator":(1,2), "item":(2,1)}
nx.draw(G, with_labels=True, pos=pos)
edges = nx.get_edge_attributes(G, 'label')
_ = nx.draw_networkx_edge_labels(G, pos=pos, edge_labels=edges)
Notes
- A generator function returns a generator when executed
- A generator "generates" items by calling
yield - How to spot a generator function?
In [29]:
def not_a_generator():
return
In [30]:
def a_generator():
yield
return
In [31]:
def also_a_generator():
if False:
yield
return
Also: execute it!
In [32]:
showfile('files/example1.py', 'files/test_example1.py')
Out[32]:
In [75]:
!py.test-3.4 files/test_example1.py
How does the execution proceed?
In [34]:
def generator_function():
print('--start--')
yield 1
print('--middle--')
yield 2
print('--stop--')
In [35]:
generator = generator_function()
In [36]:
next(generator)
Out[36]:
In [37]:
next(generator)
Out[37]:
In [38]:
next(generator)
Generator objects
next(g)callsg.next()(Python 2) org.__next__()(Python 3, 4,...)g.send()enables bidirectional communication
In [76]:
def bidirectional_generator_function():
print('--start--')
val = yield 1
print('--got', val)
print('--middle--')
val = yield 2
print('--got', val)
print('--stop--')
In [77]:
generator = bidirectional_generator_function()
In [78]:
next(generator)
Out[78]:
In [79]:
generator.send('value')
Out[79]:
In [80]:
next(generator)
Careful about the first step
In [81]:
generator = bidirectional_generator_function()
generator.send('something')
In [82]:
generator = bidirectional_generator_function()
generator.send(None)
Out[82]:
In [83]:
showfile('files/example2.py', 'files/test_example2.py')
Out[83]:
Handling exceptional circumstances
skip?
In [120]:
import math
def f(n):
print('will call math.sqrt')
ans = math.sqrt(n)
print('math.sqrt has returned')
return ans
def g(n):
print('will call f')
ans = 2 * f(n)
print('f has returned')
return ans
g(-1)
Catching exceptions¶
In [153]:
def inverse(val):
return 1/val
def g(val):
try:
return inverse(val)
except Exception as e:
print('caught exception', e.__class__.__name__, ':', e)
return 0
In [157]:
print(g(5))
print(g(0))
Doing something every time¶
In [154]:
def f(flag):
try:
if flag:
no_such_func()
return 11
except Exception as e:
print('caught exception', e.__class__.__name__, ':', e)
finally:
print('finally called')
In [155]:
f(False)
Out[155]:
In [156]:
f(True)
Explicitly raising exceptions¶
In [151]:
def sqrt(n):
if n < 0:
raise ValueError('no negativity please')
return n ** 0.5
In [152]:
print(sqrt(5))
print(sqrt(-3))
Exercise
- Write a function which uses
random.random()to decide which exception to throw (e.g.ZeroDivisionError,ValueError,Exception,TypeError, ...). - Write a
try..excepthandler which catches multiple exception types using multipleexceptclauses and prints out the exceptions.
Sending exceptions into the generator
In [84]:
def generator_function(): # same as before
print('--start--')
yield 1
print('--middle--')
yield 2
print('--stop--')
generator = generator_function()
next(generator)
Out[84]:
In [85]:
generator.throw(ZeroDivisionError)
.close() is used to destroy resources tied up in the generator
In [86]:
def generator_function():
try:
yield 'hello'
except GeneratorExit:
print('bye!')
In [87]:
g = generator_function()
next(g)
Out[87]:
In [88]:
g.close()
In [89]:
showfile('files/example3.py')
Out[89]:
Part II: decorators¶
Summary
This amazing feature appeared
in the language almost apologetically
and with concern that it might not
be that useful.
Bruce Eckel
- Decorators give us a chance to modify a function after it is defined
- Possible in Python since the beginning, but decorator syntax makes this easy
In [92]:
@deco
def function():
print('in function')
In [93]:
def function(): # old syntax before the introduction of @
print('in function')
function = deco(function)
Decorators for class functions and properties
In [94]:
class A(object):
def method(self, arg):
return arg
a = A()
print(a.method(1))
In [95]:
class A(object):
@classmethod
def cmethod(cls, arg):
return arg
a = A()
print(a.cmethod(2))
print(A.cmethod(3))
In [96]:
class A(object):
@staticmethod
def method(arg):
return arg
a = A()
print(a.method(4))
print(a.method(5))
In [97]:
class A(object):
@property
def not_a_method(self):
return 'value'
a = A()
print(a.not_a_method)
@property makes attributes read-only
In [98]:
class Square(object):
def __init__(self, edge):
self.edge = edge
@property
def area(self):
"""Computed area."""
return self.edge ** 2
s = Square(2)
s.area
Out[98]:
In [99]:
s.area = 3
@property-ies can be also be read-write
In [100]:
class Square(object):
def __init__(self, edge):
self.edge = edge
@property
def area(self):
"Compute area from edge"
return self.edge ** 2
@area.setter
def area(self, area):
"Set edge by computing from area"
self.edge = area ** 0.5
s = Square(2)
s.area
Out[100]:
In [101]:
s.area = 9
s.edge
Out[101]:
The property triple: setter, getter, deleter
- attribute access
s.edgecallsarea.getx - set with
@property
- setting
s.edgecallsarea.setx - set with
@area.setter
- deleting
s.edgecallsarea.delx - set with
@area.deleter
Implementing decorators
A decorator can be used to:
do stuff when a function is created
or replace a function (with a wrapper)
Do something when a function is defined
(this means that the decorator returns the argument it received)
In [102]:
FUNCTIONS_DEFINED_BY_ORDER = []
def remember_this_function(f):
print('just decorating', f)
FUNCTIONS_DEFINED_BY_ORDER.append(f)
return f
In [103]:
@remember_this_function
def my_func():
print('just doing my job')
In [104]:
my_func()
FUNCTIONS_DEFINED_BY_ORDER
Out[104]:
In [105]:
showfile('files/decorator_attribute.py')
Out[105]:
Replacing the function, a.k.a. doing something every time
Example: log the arguments and return value whenever a function is called
In [106]:
def logargs(f):
def wrapper(*args, **kwargs):
print('-- calling', f.__name__, 'with', args, 'and', kwargs)
ans = f(*args, **kwargs)
print('-- calling', f.__name__, 'results in', ans)
return ans
return wrapper
In [107]:
@logargs
def thinker(x, y, z):
"This function ponders for a moment and then returns the result"
print('let me think')
return x + ' ' + y + ' ' + z
thinker('mala', 'kava', z='s mlijekom')
Out[107]:
In [108]:
showfile('files/deprecation.py')
Out[108]:
The "docstring problem"
- Our function is replaced by the wrapper which is "anonymous"
- we lost the docstring
- the attributes as missing
- the signature is generic
In [109]:
print(thinker)
help(thinker)
In [110]:
%%python3
def logargs(f):
def wrapper(*args, **kwargs):
print('-- calling', f.__name__, 'with', args, 'and', kwargs)
ans = f(*args, **kwargs)
print('-- calling', f.__name__, 'result is', ans)
return ans
return wrapper
@logargs
def f():
1/0
f()
Copy the docstring and other attributes of the original
__doc____module__and__name____dict__evalis required for the rest :(- module
decoratorcompiles functions dynamically
In [111]:
import functools
def logargs(f):
def wrapper(*args, **kwargs):
print('-- calling', f.__name__, 'with', args, 'and', kwargs)
ans = f(*args, **kwargs)
print('-- calling', f.__name__, 'result is', ans)
return ans
return functools.update_wrapper(wrapper, f)
# This ^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^ is new
In [112]:
@logargs
def f():
"This function doesn't do anything really."
print('here')
f()
In [113]:
print(f)
help(f)
Exercise n
Modify decorator from previous exercise to use functools.update_wrapper or functools.wraps.
Decorators which take arguments
In [115]:
@deprecated('use math.sqrt instead')
def mysqrt(n):
"""Compute the square root using Heron's method.
Taken from http://stackoverflow.com/posts/12851282/revisions
"""
if n == 0:
return 0
if n < 1:
return mysqrt(n * 4) / 2
if 4 <= n:
return mysqrt(n / 4) * 2
x = (n + 1) / 2
for i in range(5):
x = (x + n/x) / 2
x = (x + n/x) / 2
return x
print('5:', mysqrt(5))
print('3:', mysqrt(3))
In [116]:
import functools
import math
def deprecated(message):
"Create a decorator which will emit message."
print('creating decorator')
def decorator(func):
"Decorate a function to emit the message on first call."
print('decorating function', func.__name__)
func.warning_seen = False
def wrapper(*args, **kwargs):
print('the wrapper for', func.__name__, 'is running')
if not func.warning_seen:
print('{} is depracted: {}'.format(f.__name__, message))
func.warning_seen = True
return func(*args, **kwargs)
return functools.update_wrapper(wrapper, func)
return decorator
In [117]:
@deprecated('use math.sqrt instead')
def mysqrt(n):
print('mysqrt is running')
return math.sqrt(n)
In [118]:
print('5:', mysqrt(5))
print('3:', mysqrt(3))
Decorators work for classes too
- same principle
- much less exciting
In [119]:
def deco(f):
print(f.__name__, 'just got defined')
return f
@deco
class A(object):
def __init__(self):
print(self.__class__.__name__, 'object is created')
print('A:', A)
In [147]:
a = A()
print('a:', a)
Example (skipped)
See Flask for a beautiful application of this mechanism.
Decorators — summary:
- decorators work for functions and classes
- decorators can return the original object or a replacement
- decorators are usually implemented as functions, but could be classes
functoolshelp preserve__doc__,__name__,__module__- nested functions are useful to hold state
Part III: context managers¶
How to make sure resources are freed?
In [127]:
import os
f = open('temporary_file', 'w')
try:
f.write('blah blah')
# do some processing using the temporary file
finally:
os.unlink(f.name)
General pattern
- Acquire resources
tryto use the resourcesfinallycleanup
Context managers
In [129]:
manager = Manager()
with manager as var:
do_something(var)
In [130]:
manager = Manager()
var = manager.__enter__()
try:
do_something(var)
finally:
manager.__exit__(...)
Defining context managers using generator functions
In [131]:
import contextlib
@contextlib.contextmanager
def some_generator(*args):
print('doing setup')
try:
yield 'value-of-variable'
finally:
print('doing cleanup')
pass
A context manager defined using a generator function¶
In [132]:
import os
@contextlib.contextmanager
def temporary_file(name):
f = open(name, 'w')
try:
yield f
finally:
os.unlink(f.name)
In [134]:
showfile('files/cm_timer.py')
Out[134]:
Nested try..finally calls
In [136]:
try:
try:
print('working')
buggy_code()
print('work done')
finally:
print('finalizer a')
buggy_cleanup()
print('cleanup done')
finally:
print('finalizer b')
Nested context managers
In [137]:
@contextlib.contextmanager
def cm(name):
print('--entering', name)
try:
yield
finally:
print('--exiting', name)
with cm('outer'), cm('inner'):
print('real stuff')
Objects can be their own managers
Context manager must have __enter__() and __exit__().
- an object has
__enter__()and__exit__()methods __enter__()returnsself
In [138]:
f = open('some-file')
help(f.__enter__)
help(f.__exit__)
In [139]:
with open('/etc/fstab', 'rt') as f:
with open('/tmp/g', 'wt') as g:
g.write(f.read())
In [140]:
f.closed, g.closed
Out[140]:
Handling exceptions in the context manager¶
In [142]:
showfile('files/cm_assert_raises.py')
Out[142]:
Summary
- function/subroutine — inner block of code
- decorator — function intro and outro
- context manager —
try..except..finally..elsecleanup - generators & iterators & generator expressions — loop control
In [143]:
IPython.display.Image('files/tux-asleep-2.png')
Out[143]:
In [144]:
showfile('files/capture_stdout.py')
Out[144]:
In [145]:
showfile('files/test_example3.py')
Out[145]:
In [146]:
showfile('files/memoize.py', 'files/test_memoize.py', 'files/test_memoize_mutable.py')
Out[146]: