Understanding Transducers Through Python – Transducers are a new and interesting functional programming concept that comes from the world of Clojure. In this talk we’ll learn about transducers by seeing how to implement them in Python. By using transducers to build familiar functional programming elements like map and filter, we’ll see that transducers are actually simple, elegant, and quite powerful.

1. @sixty_north
Understanding Transducers
Through Python
1
Austin Bingham
@austin_bingham
Sunday, January 25, 15

2. 237
“Transducers are coming”
Rich Hickeyhttp://blog.cognitect.com/blog/2014/8/6/transducers-are-coming
Photo: Howard Lewis Ship under CC-BY
Sunday, January 25, 15

3. Functions which transform reducers
What is a transducer?
‣ Reducer (reducing function)
• Any function we can pass to reduce(reducer, iterable[, initial])
• (result, input) → result
• add(result, input)
reduce(add, [1, 2, 3], 0) → 6
‣ Transducer (transform reducer)
• A function which accepts a reducer, and transforms it in some way, and returns a
new reducer
• ((result, input) → result) → ((result, input) → result)
3
Sunday, January 25, 15

4. Transducers are a functional programming technique not restricted to Clojure
How does this relate to Python?
‣ Clojure implementation is the prototype/archetype
• Heavily uses anonymous functions
• Uses overloads on function arity for disparate purposes
• Complected with existing approaches
‣ Python implementation is pedagogical (but also useful)
• Explicit is better than implicit
• Readability counts!
• Has all the functional tools we need for transducers
• I’m a better Pythonista than Clojurist
4
Sunday, January 25, 15

5. 5
Review
functional programming
tools in
Sunday, January 25, 15

6. 6
my_filter(is_prime, my_map(prime_factors, range(32) ) )
iterable
sequence
sequence
Sunday, January 25, 15

7. 7
def my_map(transform, iterable):
def map_reducer(sequence, item):
sequence.append(transform(item))
return sequence
return reduce(map_reducer, iterable, [])
def my_filter(predicate, iterable):
def filter_reducer(sequence, item):
if predicate(item):
sequence.append(item)
return sequence
return reduce(filter_reducer, iterable, [])
Sunday, January 25, 15

8. 7
def my_map(transform, iterable):
def map_reducer(sequence, item):
sequence.append(transform(item))
return sequence
return reduce(map_reducer, iterable, [])
def my_filter(predicate, iterable):
def filter_reducer(sequence, item):
if predicate(item):
sequence.append(item)
return sequence
return reduce(filter_reducer, iterable, [])
reduce()
Sunday, January 25, 15

9. 7
def my_map(transform, iterable):
def map_reducer(sequence, item):
sequence.append(transform(item))
return sequence
return reduce(map_reducer, iterable, [])
def my_filter(predicate, iterable):
def filter_reducer(sequence, item):
if predicate(item):
sequence.append(item)
return sequence
return reduce(filter_reducer, iterable, [])
reduce()
sequence.append()
Sunday, January 25, 15

10. 7
def my_map(transform, iterable):
def map_reducer(sequence, item):
sequence.append(transform(item))
return sequence
return reduce(map_reducer, iterable, [])
def my_filter(predicate, iterable):
def filter_reducer(sequence, item):
if predicate(item):
sequence.append(item)
return sequence
return reduce(filter_reducer, iterable, [])
reduce()
sequence.append()
Empty list : ‘seed’
must be a mutable
sequence
Sunday, January 25, 15

11. 8
>>> reduce(make_mapper(square), range(10), [])
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>> reduce(make_filterer(is_prime), range(100), [])
[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61,
67, 71, 73, 79, 83, 89, 97]
‣ make_mapper() and make_filterer() are not composable
‣ to square primes we would need to call reduce() twice
‣ this requires we store the intermediate sequence
‣ the reducers returned by make_mapper() and make_filterer()
depend on the seed being a mutable sequence e.g. a list
Sunday, January 25, 15

12. 9
(defn map
([f]
(fn [rf]
(fn
([] (rf))
([result] (rf result))
([result input]
(rf result (f input)))
([result input & inputs]
(rf result (apply f input inputs))))))
Sunday, January 25, 15

13. 10
(defn map ;; The tranducer factory...
([f] ;; ...accepts a single argument 'f', the transforming function
(fn [rf] ;; The transducer function accepts a reducing function 'rf'
(fn ;; This is the reducing function returned by the transducer
([] (rf)) ;; 0-arity : Forward to the zero-arity reducing function 'rf'
([result] (rf result)) ;; 1-arity : Forward to the one-arity reducing function 'rf'
([result input] ;; 2-arity : Perform the reduction with one arg to 'f'
(rf result (f input)))
([result input & inputs] ;; n-arity : Perform the reduction with multiple args to 'f'
(rf result (apply f input inputs))))))
Sunday, January 25, 15

14. 11
(defn map ;; The tranducer factory...
([f] ;; ...accepts a single argument 'f', the transforming function
(fn [rf] ;; The transducer function accepts a reducing function 'rf'
(fn ;; This is the reducing function returned by the transducer
([] (rf)) ;; 0-arity : Return a 'seed' value obtained from 'rf'
([result] (rf result)) ;; 1-arity : Obtain final result from 'rf' and clean-up
([result input] ;; 2-arity : Perform the reduction with one arg to 'f'
(rf result (f input)))
([result input & inputs] ;; n-arity : Perform the reduction with multiple args to 'f'
(rf result (apply f input inputs))))))
Sunday, January 25, 15

15. 12
(defn map ;; The tranducer factory...
([f] ;; ...accepts a single argument 'f', the transforming function
(fn [rf] ;; The transducer function accepts a reducing function 'rf'
(fn ;; This is the reducing function returned by the transducer
([] (rf)) ;; 0-arity : Return a 'seed' value obtained from 'rf'
([result] (rf result)) ;; 1-arity : Obtain final result from 'rf' and clean-up
([result input] ;; 2-arity : Perform the reduction with one arg to 'f'
(rf result (f input)))
([result input & inputs] ;; n-arity : Perform the reduction with multiple args to 'f'
(rf result (apply f input inputs))))))
To fully implement Clojure’s transducers in Python we also need:
‣ explicit association of the seed value with the reduction operation
‣ support for early termination without reducing the whole series
‣ reduction to a final value and opportunity to clean up state
Sunday, January 25, 15

16. 13
class Reducer:
def __init__(self, reducer): # Construct from reducing function
pass
def initial(self): # Return the initial seed value
pass # 0-arity
def step(self, result, item): # Next step in the reduction
pass # 2-arity
def complete(self, result): # Produce a final result and clean up
pass # 1-arity
Sunday, January 25, 15

17. 13
class Reducer:
def __init__(self, reducer): # Construct from reducing function
pass
def initial(self): # Return the initial seed value
pass # 0-arity
def step(self, result, item): # Next step in the reduction
pass # 2-arity
def complete(self, result): # Produce a final result and clean up
pass # 1-arity
new_reducer = Reducer(reducer)
Sunday, January 25, 15

18. 13
class Reducer:
def __init__(self, reducer): # Construct from reducing function
pass
def initial(self): # Return the initial seed value
pass # 0-arity
def step(self, result, item): # Next step in the reduction
pass # 2-arity
def complete(self, result): # Produce a final result and clean up
pass # 1-arity
new_reducer = Reducer(reducer)
def transducer(reducer):
return Reducer(reducer)
Sunday, January 25, 15

19. 13
class Reducer:
def __init__(self, reducer): # Construct from reducing function
pass
def initial(self): # Return the initial seed value
pass # 0-arity
def step(self, result, item): # Next step in the reduction
pass # 2-arity
def complete(self, result): # Produce a final result and clean up
pass # 1-arity
new_reducer = Reducer(reducer)
def transducer(reducer):
return Reducer(reducer)
⇐ two names for two concepts
Sunday, January 25, 15

20. Python Transducer implementations
14
Cognitect
https://github.com/cognitect-labs
PyPI: transducers
• “official”
• Python in the style of Clojure
• Only eager iterables (step back from
regular Python)
• Undesirable Python practices such as
hiding built-ins
• Also Clojure, Javascript, Ruby, etc.
Sixty North
http://code.sixty-north.com/python-
transducers
PyPI: transducer
• Pythonic
• Eager iterables
• Lazy iterables
• Co-routine based ‘push’ events
• Pull-requests welcome!
Sunday, January 25, 15

21. 15
Thank you!
@sixty_north
Austin Bingham
@austin_bingham
http://sixty-north.com/blog/
Sunday, January 25, 15