Having programmers do data science is terrible, if only everyone else were not even worse. The problem is of course tools. We seem to have settled on either: a bunch of disparate libraries thrown into a more or less agnostic IDE, or some point-and-click wonder which no matter how glossy, never seems to trully fit our domain once we get down to it. The dual lisp tradition of grow-your-own-language and grow-your-own-editor gives me hope there is a third way. This presentation is a meditation on how I approach data problems with Clojure, what I believe the process of doing data science should look like and the tools needed to get there. Some already exists (or can at least be bodged together); others can be made with relative ease (and we are already working on some of these); but a few will take a lot more hammock time. Talk delivered at :clojureD 2016 http://www.clojured.de/

1. Doing data science
with Clojure
@sbelak
simon@goopti.com

4. Design
constraints

5. The analytics chasm
Ideal. Almost real-time, can
be done during brainstorming
without disrupting flow
< 2min < 20min project
squeeze in
somewhere
in the day
fail
roadmap
ahoy!

6. Think in distributions,
not numbers

7. No
throwaways

8. Sharing results
• Have one canonical version that is always current.
• Concentrate discussion in one place and make it
searchable and persistent.
• Include methodology (=code).

9. The environment

10. REPL vs. notebook

11. REPL vs. notebook+

13. (hacked) gorilla-repl.org
+
auto-refresh
+
hypothes.is

14. #alderaan #sales #growth

15. Code hidden, but
can be expanded
Questions,
comments,
&
annotations
Shareable
Periodically re-run
to keep it fresh
#alderaan #sales #growth
discoverability

16. Wishlist/TODO
• Better editor (shaunlebron.github.io/parinfer/ ?)
• Embedded REPL
• Better exception reporting
• Browsable data structures
(tried and miserably failed: org-babel)

17. The tools

19. Data frame
• Data tends to be heterogeneous
• Clojure excels in structure manipulation/encoding

20. github.com/sbelak/huri
• No data structures, just functions over collections
• Composable (even DSLs — no macros!)
• Reasonably fast (transducers <3)
• Do-what-I-mean (auto-sort, liberal with inputs, …)
• Minimal buy-in
• Support reaching into nested structures everywhere

21. composable
data structure
based DSLs
->> and partial friendly
Support reaching into
nested structures
everywhere
vanilla vector of maps
interoperability
Provide curried versions
where possible

22. Composability is key to
quick iterating
• Provide curried versions where possible
• ->> and partial friendly
• encode computation in structure (comp, some-fn,
every-pred, data structure based DSLs, …)
• consistent API

23. Catching errors early more context
easier debugging faster iterating

24. <3 Bret Victor

25. Q: What about machine learning?
A: farm it out to
sklearn

26. huri.plot
• DSL on top of ggplot2 (via gg4clj)
• Targets Gorilla REPL
• Follows the rest of Huri’s design philosophy
• bar chart, scatter plot, line chart, box & violin plot,
heatmap, histogram

28. Wishlist/TODO
• (even) better structure manipulation (via Spectre?)
• Interactive plots
• More transducer-compatible (online) math
functions
• Optimizing ->> (rewrite code on the fly to do more
with transducer composition)

29. Projects worth keeping
an eye on
github.com/thi-ng/geom
github.com/yieldbot/vizard
zeppelin-project.org
github.com/aphyr/tesser
github.com/nathanmarz/specter

30. Questions
@sbelak
github.com/sbelak/huri