Simple introduction to Datomic

1. Introduction to Datomic
Siva Jagadeesan

2. “there’s an another
database!”
• what do you think of first?

3. The job of a database
•Coordination
•Consistency
•Indexing
•Storage
•Queries

4. The traditional world

5. • Place-oriented-programming (PLOP)
• designed decades ago
• tiny RAM, tiny disks
• Collocated components
• Difficult to scale
• Necessitates application-level sharding
The dreaded update-in-place

6. sharding
• lose
• consistency
• transactions
• queries

7. key-value stores
• lose
• querying
• really just storage systems
• not “databases”

8. Deconstruction
• relocate subsystems
• separate services each do one thing
• simplification

9. Datomic architecture
• Peers
• Transactors
• Storage services

10. The new world

11. Benefits
• Separating reads and writes
• applications only read from storage
• transactor
• handles transactions
• provides consistency
• reflects changes to peers

12. Benefits
• Integrated data distribution
• built-in in-memory caches
• self-tunes to working set
• automatic

13. Benefits
• Peers each have a query engine
• Datalog
• simple rules and data patterns
• declarative: implicit joins
• declarative: meaning is evident
• locality: datomic db and app data

14. Benefits
• Elasticity
• as elastic as peers

15. Benefits
• Cloud ready
• commodity hardware
• resilient to failures

16. Data model
• Immutable data
• remember everything
• things don’t actually “change”
• audit everything
• automatic

17. Data model
• Facts-oriented (vs. PLOP)
• Atomic Data
• datoms
• entity, attribute, value, transaction
• Not embedded into strucures

18. Data model
• Minimal Schema
• at the datom level
• directly supports cardinality
• avoid rigidity
• even other “schema-free” documents
impose their structure into apps
• Hickey:“nothing pivots like a datom”

19. Programming model
• Peer embedded in your application
• Pulls indexes or data segments as needed
• Caches locally, gets updates from transactor

20. Programming model
• No strings-based query language
• Data-structure-driven
• Lists and maps
• Easy to generate and test

21. Programming model
• Transactional
• Totally ordered
• First class: peers get a queue of all transactions
• Can use them in queries
• Facilitates event-driven triggers without polling
• Annotated transactions

22. Programming model
• Datalog
• deductive query system
• facts = datomic db
• can also include other app data
• extensible through custom functions that
queries can use

23. Programming model
• Consistent
• without impeding other threads or peers
• reads/queries without transactions
• through immutability
• db as a value

24. Programming model
• Time
• as-of queries
• windowed queries

25. The so-what
• Simplicity
• consistency
• built-in caching
• no manual sharding
• configuration-free
• cloud-scale

26. The so-what
• Evolvability
• datom-level schema
• application free to change “structural”
thinking

27. The so-what
• Local query power
• each peer has full query engine
• the db is effectively local
• isolation from others:
• e.g. - analytics usage won’t bog down
transaction processing

28. The so-what
• Multiple storage choices
• in-memory for developing, unit-testing
• RDBMS backend possible (behind firewall,
etc)
• Distributed storage services for
redundancy and unlimited read scaling
• backup/restore tools available

29. The so-what
• Integrated mem-cache support
• local caches can be backed by OTS mem-
cache cluster
• datomic is a “good citizen” user

30. The so-what
• Leverage datalog against multiple sources
• datomic db
• application data

31. The so-what
• Travel through time
• audits
• bug-resolution

32. Schema def
{:db/id {:part :db.part/db, :idx -1000001},
:db/ident :user/login,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/fulltext true,
:db/doc "the login handle of a user",
:db.install/_attribute :db.part/db}

33. Schema def
{:db/id {:part :db.part/db, :idx -1000003},
:db/ident :user/friends,
:db/valueType :db.type/ref,
:db/cardinality :db.cardinality/many,
:db/fulltext false,
:db/doc "the friends of a user",
:db.install/_attribute :db.part/db}

34. transactions
[:db/add entity-id attribute value]
[:db/retract entity-id attribute value]
{:db/id entity-id!
attribute value!
attribute value!
... }
[:db.fn/retractEntity entity-id]

35. identities
{:part :db.part/user, :idx -1000026}
{:db/id 17592186045420}

36. queries
[:find ?e :in $ ?email !
:where !
[?e :person/email ?email]]
[:find variables :where clauses]
[:find ?e ?x !
:where !
[?e :age 42] [?e :likes ?x]]

37. queries
• lots more
• pattern matching
• logic programming
• multiple data-sources

38. demonic
• datomic helper
• work at the data-structure level
• supports graphs: nested maps
• only writes dirty data
• “demarcations” as “batched datomic
transactions”
• unit-testing support

39. Questions