Using Clojure, NoSQL Databases and Functional-Style JavaScript to Write Gext-Generation HTML5 Apps

Mallorca, 24 September 2010 Presentation @ HackFwd Build 03, Release 0.1 TheDeadline GmbH & Co. KG Using Clojure, NoSQL Databases and Functional-Style JavaScript to Write Next- Generation HTML5 Apps

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 2 Since 1999 I have been running a successful software development company. In 2009 we started building TheDeadline with Clojure. We are a HackBox! And I am a Board Member of HackFwd: Eat your own dogfood! > whoami Stefan Richter Apple II Pascal, C64, 6502 VAX, Unix, C, 4GLs, Smalltalk Lisp, Scheme Unix, C/C++, 4 GLs, Windows NT, VB (!) University studies and work as a programmer in research institutes Working as a professional programmer Running my own company 20102005199819951990198719831966 Linux, Java, Python, Ruby Objective-C, C/C++ Common Lisp, Clojure, JavaScript

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Please pardon the colors. 3

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 The Web is dead. 4

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Okay. We all know that more and more people are accessing the internet via mobile devices (smartphones). 5

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 By accident, Apple ”invented“ the app economy. 6

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 People are buying specialized apps which are bound to a specific platform (iPhone/iPad, Android, etc.). 7

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 The Wired magazine calls these platforms ”walled gardens.“ 8

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Because: These apps use the internet for data transmission and networking, but they are not on the web. 9

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 It seems like people are building more and more stuff for ”walled gardens“!? 10

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Do we really have to build for ”walled gardens“? 11

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 No. Because it is a waste of time to build apps that run on just one platform. 12

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 And it is a waste of time to implement the same app for different platforms. 13

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 We (TheDeadline-Team) had the same problem: We needed mobile clients for iPhone/iPad and Android. 14

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 But then we would have to maintain at least three different platforms: Web, iPhone, Android. 15

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 So, every feature would require three times the effort. 16

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Or: Our team would become three times slower. 17

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Our solution: Build an unified HTML5 client that runs in all modern web browsers as well as on both iPhone and on Android. 18

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 As you all know, HTML5 is basically a set of browser APIs and HTML extensions. Some of them can be used to build a web client that feels like a native client. 19

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 20

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 HTML5 can do much more (video, 2D canvas, etc.), but we don’t need it for our core app. 21

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 What an ”HTML5 client“ is depends on the type of app you are building. 22

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 My definition of an HTML5 client: - App runs mostly inside the browser (JavaScript) - App can be used offline - Syncs data automatically with server - Feels more like a native app, but no WIMP interface (use CSS3) - Supports only modern browsers, no IE6/IEx fallbacks ;) 23

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 So, what parts of the HTML5 standard do we need to implement this? We need: - Client-side storing of data - Client-side caching of UI - Optional: - New stylable HTML5 elements - CSS3 media queries 24

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 We are using the Local Storage API to store all user data locally on the client side. 25

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 We are using the Caching API to cache the static UI parts on the browser-side. 26

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 27 Google App Engine Clojure AppEngine SDK for Java Datastore TaskQueues Compojure Browser JavaScript JQuery (mostly) Before: a request/response style web-application. HTTP/SSL HTML/CSS/JS

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Almost everything was rendered on the server. (Okay, we had some AJAX, too. But not much.) 28

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 For the server-side rendering, we are using Compojure and Hiccup. 29

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 On the next slide, you can see how nice it is to have your application code, HTML and CSS in one place. It is also a good example of how to use multimethods for UI rendering. (You don‘t need a web-framework when you are using Lisp). 30

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 You see, we are not only strong supporters of the NoSQL movement, we want to suggest a new movement: The No Templates movement. ;) 32

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Too bad that we can’t use this on the client side. So, how can we render the UI on the client side? 33

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Most people would shout: JQuery! 34

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 The idea behind JQuery is to execute (chained) function calls on a set of DOM elements. This is why JQuery is so easy to learn and to use. You don’t even have to know how to program JavaScript! 35

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 But can you build a large HTML5 client that consists solely of JQuery plugins which just operate on the DOM-tree? 36

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 You will end up with lots of code blocks that all iterate over the same DOM tree. For a large client, we think you need to bring more structure to your code. 37

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 So we tried Google Closure Library. Google Closure Libary is to JavaScript what the JDK is to Java: a mature standard library. 38

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 And it helps you to structure your code by providing a namespace system. 39

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 BTW, you can’t compare JQuery and Google Closure using simple examples. It isn’t a question of writing five lines of elegant code. The question is wether you can write thousands of lines of complex code ... 40

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 I have to admit: 10 years ago I hated JavaScript. The code we received from web agencies was such a mess. It was slow. The JavaScript interpreters inside the browser were broken. It was impossible to see the beauty behind the beast ... 41

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Today most people are trying to structure JavaScript code like Java code. 42

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 I think this is wrong: JavaScript may look better now, but it is bloated in the same way that Java is ... 43

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Let’s stop the bloat: We are already using a functional programming language on the server side (Clojure). Of course, we want to program in the same style on the client side! 44

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 What do I mean by ”functional programming“*? - First class functions - Just returning values - Causing no side effects - Functions as parameters - Function combination *(Yes, this definition is too simple) 45

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 JavaScript makes it all possible. 46

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 And: Google Closure comes with a library of functions that enables you to work with collections of data in a functional programming style. 47

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Check goog.functions: 48

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Check goog.array: map, reduce, every, filter, flatten, some, repeat, etc. These functions don’t alter their input. 50

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Implementation started ... 51

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 First we implemented a key/ value store API on top of the HTML5 Local Storage API. It behaves very much like the Google App Engine Datastore. We can store arbitrary maps identified by a :key and a :kind. 52

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 This key/value store on the client side (we call it event- db) automatically syncs with the Google datastore on the server. Like this: 53

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 54 Google App Engine Browser Auto-Sync event-db API XHR/JSON Notification Datastore Local Storage API TheDeadline HTML5 UI (rendering) Notification Data- Access (Maps/JS- Objects)

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 A small step for mankind, but a huge step for TheDeadline: Now we just use the event- db API to access the user data. We don’t need to worry about XHR anymore ... 55

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Cool thing: Throughout the whole TheDeadline-system we just pass around a very simple datastructure: maps of key/ value pairs. No complex DB- mappings needed. NoSQL. Just filter, map, reduce. Easy transformations. 56

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 The Google Datastore is a distributed key/value store. Clojure works with key/ value structures. JSON is build on key/value structures. 57

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 From this perspective is functional programming the opposite of object-oriented programming: - Simple data structures - Powerful functional operations on these data structures - No state (well, almost) 58

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 We try to write as much functional code as possible. But the UI must maintain some state. 59

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 So sometimes object- orientation can be useful: A Widget Library is a good example. Google Closure comes with such a library, which you can extend by building your own UI components. 60

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 When you have to encapsulate state, you can write a component. For rendering the client- side UI we are using Google Closure Templates. Like this: 61

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Google Closure Templates are used like function calls: You pass the data to be rendered as parameter. Like this: 63

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 64 The application of the Soy template:

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Just a reminder: This code runs on all platforms. Computers and smartphones. 65

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Compare this with the work you have to do when writing proprietary clients for each platform. And I can tell you: This is much more fun! 66

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Now we have a nice HTML5 client. You can use it on several devices at the same time. You can use it offline and sync later. Uhh. But what happens when you have concurrent modifications?! 67

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 You need a data model that is able to store versions of your user data! 68

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Relational databases are not well suited for data versioning. Try to version a large graph of entities distributed over several tables. Good luck! 69

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Good to have a schema-free, NoSQL, key/value store! (No matter what Michael Stonebraker says.) 70

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Our solution is almost a functional data structure: We store every change as an operation on an entity. We can replay all operations. We could easily provide undo-operations. We can restore every state change. You can see it in our todo history: 71

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 72 The history shows all operations that were applied to a todo entity.

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 A schema-free database like Google’s datastore helps a lot to implement such a model. 73

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 And to make and keep the code really elegant, a dynamic, functional programming language like Clojure helps, too! 74

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 What else are we doing on the server side with Clojure? 75

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 We are trying to make modern applications more intelligent. 76

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 TheDeadline contains a rule-based expert system. 77

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 We are using the expert system for decision support: We want to provide the right information to the right person at the right time in real-time in a project management situation. 78

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 When you have lots of rules, you don’t want to make decisions in cascading if- then-else clauses. 79

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 The expert system is a Clojure-based implementation of a classical production system like OPS5 or CLIPS. 80

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 So, on the server side we are processing large datasets: We generate facts from the user data and run them through our expert- system, which then generates actions for the users based on our rule-set. 82

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 And again: Clojure really helps us to implement stuff like this. 83

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Conclusion 84

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 So when you want to start a company, it is very likely, that you have to build for mobile platforms from the beginning. Like Eric Schmidt said: Mobile first. 85

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Instead of building several apps for proprietary platforms use HTML5 as an open standard. (This saves time and money, too). 86

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 If you want to be successful, stay away from relational databases. Use schema-free databases to build versioning data structures. You will need them to resolve concurrent modifications of user data. 87

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Become a JavaScript master. Use a mature library. Your HTML5 client will become big and complex. 88

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 Start thinking in a functional way. Not everything is an object... 89

TheDeadline GmbH & Co. KG Presentation @ HackFwd Build 03 Release 0.1 And: Have fun! Starting a company can be so much fun. And maybe you sign-up for HackFwd?! I‘d love to see more Clojure Hackers building their dreams! 90

Thank you. TheDeadline: http://www.the-deadline.com Corporate Blog: http://www.hackers-with-attitude.com Corporate WebSite: http://www.freiheit.com Self: http://www.smartrevolution.de http://www.twitter.com/smartrevolution