Having programmers do data science is terrible, if only everyone else were not even worse. The problem is of course tools. We seem to have settled on either: a bunch of disparate libraries thrown into a more or less agnostic IDE, or some point-and-click wonder which no matter how glossy, never seems to truly fit our domain once we get down to it. The dual lisp tradition of grow-your-own-language and grow-your-own-editor gives me hope there is a third way. This presentation is a meditation on how I approach data problems with Clojure, what I believe the process of doing data science should look like and the tools needed to get there. Some already exist (or can at least be bodged together); others can be made with relative ease (and we are already working on some of these); but a few will take a lot more hammock time. Clojure is fantastic for data manipulation and rapid prototyping, but falls short when it comes to communicating your insights. What is lacking are good visualization libraries and (shareable) notebook-like environments. I'll show my workflow in org-babel which weaves Clojure with R (for ggplot) and Python (for scikit-learn) and tell you why it's wrong, how IPythons of the world have trapped us in a local maximum and how we need a reconceptualization similar to what a REPL does to programming. All this interposed with my experience doing data science with Clojure (everything from ETL to on-the-spot analysis during a brainstorming).

1. simon@metabase.com
@sbelak
Doing data science with
Clojure

2. My perspective
• Core developer at Metabase, an open source BI/analytics
tool. 3rd largest BI tool in the world. 21k+ companies use
us daily, including N26, Revolut, Swisscom, Confluent, …
• Built analytics department at GoOpti from the ground up
• Helped 20+ companies become data-driven

3. Goal: answer 80% of
questions stemming from
data in <20min

4. The analytics chasm
2 min 20 min project
Ideal. Almost real-time. Can be
done during brainstorming
without disrupting the flow.
:(
Added to roadmapSqueeze in
somewhere
in the day

5. Doing our job better is
often a matter of speed
and quantity

7. Clojure at a glance
• (lisp (running-on :JVM))
• Functional, dynamic, immutable
• Excellent concurrency and state-management primitives
• Unparalleled data manipulation

8. Affordances of our tools
shape how we approach
problems

9. Data frames considered
harmful
• Data frame (=table) conflates representation and
abstraction
• Clojure excels in structure manipulation/encoding

10. Composability is key to
quick iterating
• Curried versions where possible
• ->> and partial friendly
• Side benefit: consistent API

11. “This is possibly Clojure’s most important property: the
syntax expresses the code’s semantic layers. An
experienced reader of Clojure can skip over most of the
code and have a lossless understanding of its high- level
intent.”
— Z. Tellman, Elements of Clojure

12. Live programming

13. Catching errors early more
context easier debugging
faster iterating

14. The power of
sharing runtime

15. clojure.spec

16. Queryable data
descriptions

17. The ecosystem

18. What about ML?
Farm it out!
(to sklearn)

19. Mini compilers targeting
a specific library in
another language

20. huri.plot
• DSL that compiles to ggplot2
• Targets Gorilla REPL
• bar chart, scatter plot, line chart, box & violin plot,
heatmap, histogram

22. Takeouts
• Speed-of-answer matters
• You don’t have to reinvent every wheel in Clojure
• Clojure is fantastic at structure manipulation, play to its
strengths
• Blurring the line between environment and work is
a powerful idea

23. Questions
P.S. We’re hiring!

24. github.com/metabase/metabase
github.com/clojupyter/clojupyter
github.com/uncomplicate/neanderthal
github.com/scicloj
github.com/sbelak/huri