Slides from January 22nd 2013

1. An Introduction to
Neo4j
Michal Bachman
@bachmanm

2. Roadmap
• Intro to NOSQL
• Intro to Graph Databases
• Intro to Neo4j
• A bit of hacking
• Current research
• Q&A
@bachmanm

3. Not Only SQL
@bachmanm

4. Why NOSQL now?
Driving trends
@bachmanm

5. Trend 1: Data Size
@bachmanm

6. Trend 2: Connectedness
GGG
Onotologies
RDFa
Folksonomies
Information connectivity
Tagging
Wikis
UGC
Blogs
Feeds
Hypertext
Text
Documents
@bachmanm

7. Trend 3: Semi-structured Data
@bachmanm

8. Trend 4: Application Architecture (80’s)
Application
DB
@bachmanm

9. Trend 4: Application Architecture (90’s)
App App App
DB
@bachmanm

10. Application Application Application
DB DB DB
@bachmanm

11. Side note: RDBMS performance
Salary List
@bachmanm

12. Four NOSQL Categories
@bachmanm

13. Key-Value Stores
• “Dynamo: Amazon’s Highly Available Key-
Value Store” (2007)
• Data model:
– Global key-value mapping
– Big scalable HashMap
– Highly fault tolerant (typically)
• Examples:
– Riak, Redis, Voldemort
@bachmanm

14. Pros and Cons
• Strengths
– Simple data model
– Great at scaling out horizontally
• Scalable
• Available
• Weaknesses:
– Simplistic data model
– Poor for complex data
@bachmanm

15. Column Family (BigTable)
• Google’s “Bigtable: A Distributed Storage
System for Structured Data” (2006)
• Data model:
– A big table, with column families
– Map-reduce for querying/processing
• Examples:
– HBase, HyperTable, Cassandra
@bachmanm

16. Pros and Cons
• Strengths
– Data model supports semi-structured data
– Naturally indexed (columns)
– Good at scaling out horizontally
• Weaknesses:
– Unsuited for interconnected data
@bachmanm

17. Document Databases
• Data model
– Collections of documents
– A document is a key-value collection
– Index-centric, lots of map-reduce
• Examples
– CouchDB, MongoDB
@bachmanm

18. Pros and Cons
• Strengths
– Simple, powerful data model (just like SVN!)
– Good scaling (especially if sharding supported)
• Weaknesses:
– Unsuited for interconnected data
– Query model limited to keys (and indexes)
• Map reduce for larger queries
@bachmanm

19. Graph Databases
• Data model:
– Nodes with properties
– Named relationships with properties
– Hypergraph, sometimes
• Examples:
– Neo4j (of course), Sones GraphDB, OrientDB,
InfiniteGraph, AllegroGraph
@bachmanm

20. Pros and Cons
• Strengths
– Powerful data model
– Fast
• For connected data, can be many orders of magnitude
faster than RDBMS
• Weaknesses:
– Sharding
• Though they can scale reasonably well
• And for some domains you can shard too!
@bachmanm

21. Social Network “path exists”
Performance
• Experiment:
• ~1k persons # persons query time
• Average 50 friends per Relational 1000 2000ms
database
person
Neo4j 1000 2ms
• pathExists(a,b)
Neo4j 1000000 2ms
limited to depth 4
• Caches warm to
eliminate disk IO
@bachmanm

22. Four NOSQL Categories
@bachmanm

23. What are graphs good for?
• Recommendations
• Business intelligence
• Social computing
• Geospatial
• MDM
• Systems management
• Web of things
• Genealogy
• Time series data
• Product catalogue
• Web analytics
• Scientific computing (especially bioinformatics)
• Indexing your slow RDBMS
• And much more!
@bachmanm

24. Neo4j is a Graph Database
So we need to detour through a little
graph theory
@bachmanm

25. @bachmanm

26. Meet Leonhard Euler
• Swiss mathematician
• Inventor of Graph
Theory (1736)
@bachmanm
http://en.wikipedia.org/wiki/File:Leonhard_Euler_2.jpg

27. http://en.wikipedia.org/wiki/Seven_Bridges_of_Königsberg @bachmanm

28. Property Graph Model
name: Michal Bachman
• nodes / vertices
• relationships / edges
title: Intro to Neo4j
• properties duration: 45
name: Neo4j name: NOSQL
@bachmanm

29. Graphs are very whiteboard-friendly
@bachmanm

30. @bachmanm

31. Neo4j
@bachmanm

32. 32 billion nodes
32 billion relationships
64 billion properties
@bachmanm

33. @bachmanm

34. http://opfm.jpl.nasa.gov/
@bachmanm

35. http://news.xinhuanet.com
@bachmanm

36. @bachmanm

37. @bachmanm

38. Community
Advanced
Enterprise
@bachmanm

39. How do I use it?
@bachmanm

40. Getting started is easy
• Single package download, includes server stuff
– http://neo4j.org/download/
• For developer convenience, Ivy (or whatever):
– <dependency org="org.neo4j" name="neo4j-community" rev="1.9.M04"/>
@bachmanm

41. Run it!
• Server is easy to start stop
– cd <install directory>
– bin/neo4j start
– bin/neo4j stop
• Provides a REST API in addition to the other
APIs we’ve seen
• Provides some ops support
– JMX, data browser, graph visualisation
@bachmanm

42. Embed it!
• If you want to host the database in your
process just load the jars
• And point the config at the right place on disk
• Embedded databases can be HA too
– You don’t have to run as server
@bachmanm

43. name: Phil Johnson
title: Cognitive Psychology
duration: 30 name: Michal Bachman
name: UX
title: Intro to Neo4j
duration: 45
name: Martin Macke
name: Jeremy White INTERESTED name: Neo4j name: NOSQL
@bachmanm

44. GraphDatabaseService neo = new EmbeddedGraphDatabase("/data/webexpo");
Transaction tx = neo.beginTx();
try {
Node speaker = neo.createNode();
speaker.setProperty("name", "Michal Bachman");
Node talk = neo.createNode();
talk.setProperty("title", "Intro to Neo4j");
Relationship delivers
= speaker.createRelationshipTo(talk,
DynamicRelationshipType.withName("DELIVERS"));
delivers.setProperty("day", ”Saturday");
neo.index().forNodes("people")
.add(speaker, "name", "Michal Bachman");
} finally {
tx.finish();
}
name: Michal Bachman DELIVERS title: Intro to Neo4j
day: Saturday
@bachmanm

46. @bachmanm

47. Core API
• Nodes
– Properties (optional K-V pairs)
• Relatiosnhips
– Start node (required)
– End node (required)
– Properties (optional K-V pairs)
@bachmanm

48. All Conference Topics
@bachmanm

49. name: Phil Johnson
title: Cognitive Psychology
duration: 30 name: Michal Bachman
name: UX
title: Intro to Neo4j
duration: 45
name: Martin Macke
name: Jeremy White INTERESTED name: Neo4j name: NOSQL
@bachmanm

50. All Conference Topics
Node webExpo = neo.getReferenceNode();
for (Relationship talksAt : webExpo.getRelationships(INCOMING, TALKS_AT)) {
Node speaker = talksAt.getStartNode();
for (Relationship delivers : speaker.getRelationships(OUTGOING, DELIVERS)) {
Node talk = delivers.getEndNode();
for (Relationship about : talk.getRelationships(OUTGOING, ABOUT)) {
String topicName = (String) about.getEndNode().getProperty(NAME);
//add to result...
}
}
}
-------------------
Printing all topics
All topics: development, data, advertising, education, usa, business, microsoft, webdesign, software,
responsiveness, ux, e-commerce, php, psychology, crm, api, chef, javascript, patterns, product design,
marketing, metro, social media, web, startup, analytics, lean, cqrs, node.js, branding, cloud, testing, neo4j,
rest, css, design, publishing, nosql. Took: 2 ms

51. Which talks should I attend?
@bachmanm

52. name: Phil Johnson
title: Cognitive Psychology
duration: 30 name: Michal Bachman
name: UX
title: Intro to Neo4j
duration: 45
name: Martin Macke
name: Jeremy White INTERESTED name: Neo4j name: NOSQL
@bachmanm

53. Which talks should I attend?
TraversalDescription talksTraversal = Traversal.description()
.uniqueness(Uniqueness.NONE)
.breadthFirst()
.relationships(INTERESTED, OUTGOING)
.relationships(ABOUT, INCOMING)
.evaluator(Evaluators.atDepth(2));
Node attendee =
neo.index().forNodes("people").get("name", ”Jeremy White").getSingle();
Iterable<Node> talks = talksTraversal.traverse(attendee).nodes();
//iterate over talks and print
------------------------------------------
Suggesting talks for 100 random attendees.
...
Aneta Lebedova: Measure Everything!, To the USA, The real me. Took: 1 ms
Bohumir Kubat: Beyond the polar bear, How (not) to do API, Critical interface design. Took: 1 ms
Vladimir Vales: Application Development for Windows 8 Metro. Took: 1 ms
Suggested talks for 100 random attendees in 449 ms

54. What do we have in common?
@bachmanm

55. name: Phil Johnson
title: Cognitive Psychology
duration: 30 name: Michal Bachman
name: UX
title: Intro to Neo4j
duration: 45
name: Martin Macke
name: Jeremy White INTERESTED name: Neo4j name: NOSQL
@bachmanm

56. What do we have in common?
//retrieve attendeeOne and attendeeTwo from index
int maxDepth = 2;
Iterable<Path> paths = GraphAlgoFactory
.allPaths(Traversal.expanderForAllTypes(), maxDepth)
.findAllPaths(attendeeOne, attendeeTwo);
for (Path path : paths) {
//print it
}
------------------------------------------------------------
Finding things in common for 100 random couples of attendees
...
Karel Kunc and Phil Smith:
(Karel Kunc)--[INTERESTED]-->(ux)<--[INTERESTED]--(Phil Smith),
(Karel Kunc)--[DISLIKED]-->(Be a punk consumer!)<--[DISLIKED]--(Phil Smith),
(Karel Kunc)--[DISLIKED]-->(Beyond the polar bear)<--[LIKED]--(Phil Smith),
(Karel Kunc)--[LIKED]-->(Shipito.com – business in USA)<--[LIKED]--(Phil Smith).
Took: 0 ms.
...
Found things in common for 100 random couples of attendees in 142 ms.

57. Youngsters, Y U No Like Java?
@bachmanm

58. Who is my beer mate?
myself beerMate:?
talk:?
@bachmanm

59. Who is my beer mate?
(myself) (beerMate)
(talk)
@bachmanm

60. Who is my beer mate?
start myself=node:people(name = "Emil Votruba")
match (myself)-[:LIKED]->(talk)<-[:LIKED]-(beerMate)
return distinct beerMate.name, count(beerMate)
order by count(beerMate) desc
limit 5;
@bachmanm

61. Cypher Query
start myself=node:people(name = ”Alex Smart")
match (myself)-[:LIKED]->(talk)<-[:LIKED]-(beerMate)
return distinct beerMate.name, count(beerMate)
order by count(beerMate) desc
limit 5;
@bachmanm

62. Cypher Query
start myself=node:people(name = ”Emil Votruba")
match (myself)-[:LIKED]->()<-[:LIKED]-(beerMate)
return distinct beerMate.name, count(beerMate)
order by count(beerMate) desc
limit 5;
@bachmanm

63. Who is my beer mate?
@bachmanm

64. Current Research
• Graph partitioning
• Graph analytics (“OLAP” and predictive)
• Performance improvements
• Query languages
• MVCC and single-threaded write models
• ACID (tradeoffs for weakening C and I)
• Yield and Harvest in distributed systems
• Application-level
– Recommendations
– Protein interactions
–…
@bachmanm

65. Questions?
Neo4j: http://neo4j.org
Neo Technology: http://neotechnology.com
Twitter: @bachmanm
Code: git://github.com/bachmanm/neo4j-imperial.git