Ryan Boyd, Head of Developer Relations, Neo4j

1. Neo4j Database & Graph Platform
GraphTour Boston – May 8, 2018
Ryan Boyd
@ryguyrg

2. An IT Portfolio View of Graph Technologies:
The Neo4j Graph Platform Vision

3. The Neo4j Graph Platform Vision
Graph
Transactions

4. Neo4j: Key Components

5. Key Architecture Components
1
Index-Free Adjacency
In memory and on flash/disk
2
vs
ACID Foundation
Required for safe writes
3
Full-Stack Clustering
Causal consistency
5
Graph Engine
Cost-Based Optimizer, Graph
Statistics, Cypher Runtime, …
6
Hardware Optimizations
For next-gen infrastructure
Language, Drivers, Tooling
Developer Experience,
Graph Efficiency, Type Safety
4

6. At Write Time:
data is connected
as it is stored
At Read Time:
Lightning-fast retrieval of data and
relationships via pointer chasing
Index-Free Adjacency:

7. Connectedness and Size of Data Set
ResponseTime
Relational and
Other NoSQL
Databases
0 to 2 hops
0 to 3 degrees
Thousands of connections
1000x
Advantage
Tens to hundreds of hops
Thousands of degrees
Billions of connections
Neo4j
“Minutes to
milliseconds”
“Minutes to Milliseconds” Real-Time Query Performance

8. Neo4j: Enabling the Connected Enterprise
Consumers of Connected Data
8
AI & Graph Analytics
• Sentiment analysis
• Customer segmentation
• Machine learning
• Cognitive computing
• Community detection
Transactional Graphs
• Fraud detection
• Real-time recommendations
• Network and IT operations
management
• Knowledge Graphs
• Master Data Management
Discovery & Visualization
• Fraud detection
• Network and IT operations
• Product information
management
• Risk and portfolio analysisData
Scientists
Business
Users
Applications

9. The Neo4j Graph Platform Vision
Graph
Analytics
Graph
Transactions

10. Finds the optimal path
or evaluates route
availability and quality
Evaluates how a
graph is clustered
or partitioned
Determines the
importance of distinct
nodes in the network
Neo4j Graph Algorithm Library

11. 450
400
350
300
250
200
150
100
50
0
GraphX
251
Neo4j
152
GraphX
416
Neo4j
124
Union-Find
(Connected Components)
Neo4j
Significantly
Faster!
PageRank
Graph Algorithm Performance

12. The Neo4j Graph Platform Vision
Graph
Analytics
Graph
Transactions
AI

13. The Neo4j Graph Platform Vision
Graph
Analytics
Graph
Transactions
AI
Data Integration

14. Common Integration Patterns Inside the Enterprise
From Disparate Silos
To Cross-Silo Connections
From Tabular Data
To Connected Data
From Data Lake Analytics
to Real-Time Operations

15. The Neo4j Graph Platform Vision
Graph
Analytics
Graph
Transactions
AI
Data Integration
Drivers & APIs
APPLICATIONS
DEVELOPERS

16. 16
Drivers & APIs
• Native Language Drivers
• Java
• .Net
• Python
• JavaScript
• more to come…
• Massive Community Support (Go, Ruby, R, Perl, Clojure, C/C++, …)
• Partners like GraphAware (PHP Client)

17. The Neo4j Graph Platform Vision
Graph
Analytics
Graph
Transactions
AI
Data Integration
Drivers & APIs
APPLICATIONS BUSINESS USERS
Discovery & Visualization
DATA
ANALYSTS
DEVELOPERS

18. Discovery & Visualization
18
Neo4j Browser
Custom / JS Libraries
Partner Applications

19. The Neo4j Graph Platform Vision
Graph
Analytics
Graph
Transactions
AI
Data Integration
Drivers & APIs
APPLICATIONS BUSINESS USERS
Discovery & Visualization
DATA
ANALYSTS
DEVELOPERS
Development
&
Administration
ADMINS

20. Development & Administration
Neo4j Browser Neo4j Desktop

21. The Neo4j Graph Platform Vision
Analytics
Tooling
BUSINESS USERS
DEVELOPERS
ADMINS
Graph
Analytics
Graph
Transactions
Data Integration
Discovery & Visualization
DATA
ANALYSTS
DATA
SCIENTISTS
Drivers & APIs
APPLICATIONS
AI
Development
&
Administration

22. Cypher for
Apache® Spark™

23. Cypher for
Apache® Spark™
Relational
Spark SQL
Graph

24. Composable Queries
Graph
Graph
Table
Cypher
Graph
Cypher

25. Better Access to Technology
Partner Software
New Neo4j-Provided
Capabilities
25
• Improved visibility, provisioning,
and integration of partners’
software
• Products and add-ons that satisfy
basic Neo4j project needs across
the software lifecycle
Platform Means:

26. The Neo4j Graph Platform Vision
Analytics
Tooling
BUSINESS USERS
DEVELOPERS
ADMINS
Graph
Analytics
Graph
Transactions
Data Integration
Discovery & Visualization
DATA
ANALYSTS
DATA
SCIENTISTS
Drivers & APIs
APPLICATIONS
AI
Development
&
Administration

27. (Cypher)
-[:IS]->
(Everywhere)

28. opencypher.org

29. Neo4j Product Update
Latest Innovations

30. 3.3
Latest GA Release
Recap
(Oct ‘17)

31. “Least Connected”
load balancing
Faster & more memory efficient
runtime
Batch generation of IDs
Schema operations now take local
locks
Page cache metadata moved off
heap
Native GB+ Tree numeric indexes
Bulk importer paging & memory
improvements
Dynamically reload config settings
without restarting Neo4j
Admin & Config
Storage & Indexing
Memory Management
Kernel & Transactions
Cypher Engine
Drivers & Bolt Protocol
Neo4j 3.3 Performance Improvements

32. Concurrent/Transactional Write Performance
25000
20000
15000
10000
5000
0
Neo4j 2.2 Neo4j 2.3 Neo4j 3.0 Neo4j 3.1 Neo4j 3.2 Neo4j 3.3
69%
31%
59%
38%
55%
(Simulates Real-World Workloads)

33. `
RR
RR RR
RRRRRR
READ REPLICAS
London
`
C C
RR RR RR
RRRRRR
READ REPLICAS
New York
Encryption
Multi-Data Center Clustering

34. LDAP & Active DirectoryUser & Roles
Procedure
Access Controls
Neo4j Security Foundation
Kerberos
Strong Security
Intra-Cluster
Encryption
Security
Event Logging
TLS Wire
Encryption

35. Neo4j Desktop 1.0
• Mission control for developers
• Connect to both local and remote
Neo4j servers
• Free with registration
• Includes development license for Neo4j Enterprise Edition
• Keeps you up to date with latest versions, plugins, etc.
• https://neo4j.com/download

36. What’s Next?
3.4

37. 37
Graphs Geospatial
+

38. 8
x
y
Geospatial Graph Queries
z

39. 9
“Recommend a shirt available in a store
close by in the men’s department”
Location Building
Floor
Rack

40. Scalability

41. Multi-Clustering Support for Global Internet Apps
Horizontally partition graph by domain (country, product, customer, data center)
41
Multi-tenancy
Geo Partitioning
Write Scaling
sa cluster
uk cluster
us_east cluster
hk cluster

42. Performance

43. Neo4j 3.3 Neo4j 3.4 RC1
70% Faster
250
500
0
750
1000
Neo4j 3.2
Reads – Neo4j Enterprise Cypher Runtime
10% Faster
Mixed Workload Read Benchmark

44. Supercharging Graph Writes
One Component:
80% of transactional write overhead (!!!)
?QM ,.
Index Insertion

45. ü ACID
ü Optimized for
graph
ü Fast Reads
ü ~10x Faster Writes
Neo4j GB+ Tree
Index
Label Groupings (Neo4j 3.2)
Numerics (Neo4j 3.3)
Strings (Neo4j 3.4)
QM ,.
External Index
Provider
Native Neo4j
Index
Supercharging Graph Writes

46. 6
Writes with Native String Indexes
500%
improvement
Neo4j 3.4 RC1
Performance

47. Ops & Admin

48. 8
Rolling Upgrades
3.4 3.4 3.44.0 4.0 4.0
Auto Cache Reheating
Available in Neo4j 3.4!
Upgrade to new versions
with zero downtime
Store upgrades
may require downtime but
can be done subsequently
For Restarts, Restores, and Cluster Expansion

49. 3.4
Available This Month!

50. 50

51. 51

52. How we think How we’re wired

55. 55

56. 56
Neo4j Bloom

57. Discovery & Visualization
57
Neo4j Browser
Custom / JS Libraries
Partner Applications

58. Demo Time!

62. Awesome Procedures
On Cypher

65. > 400
procedures and functions

68. Data Integration

69. Load from a relational database

70. apoc.load.jdbc
WITH "jdbc:mysql://localhost:3306/northwind?user=root" AS url
CALL apoc.load.jdbc(url,"products")
YIELD row
MERGE (p:Product {id: row.ProductID})
SET p.name = row.ProductName, p.unitPrice = row.UnitPrice

71. WITH "jdbc:mysql://localhost:3306/northwind?user=root" AS url
CALL apoc.load.jdbc(url,"products")
YIELD row
MERGE (p:Product {id: row.ProductID})
SET p.name = row.ProductName, p.unitPrice = row.UnitPrice
Execute procedure

72. Apply Cypher transformation
WITH "jdbc:mysql://localhost:3306/northwind?user=root" AS url
CALL apoc.load.jdbc(url,"products")
YIELD row
MERGE (p:Product {id: row.ProductID})
SET p.name = row.ProductName, p.unitPrice = row.UnitPrice

73. Load XML

74. apoc.load.xml
CALL apoc.load.xml('http://overpass.osm.rambler.ru/cgi/xapi_meta?*[bbox=11.54,48.14,11.543,48.145]')
YIELD value
UNWIND value["_children"] AS child
WITH child WHERE child["_type"] = "node"
WITH child.id AS id,
child.lat AS latitude,
child.lon AS longitude,
child["user"] AS userName
MERGE (point:Point {id: id})
SET point.latitude = latitude,
point.longitude = longitude
MERGE (user:User {name: userName})
MERGE (user)-[:EDITED]->(point)

75. Execute procedure
CALL apoc.load.xml('http://overpass.osm.rambler.ru/cgi/xapi_meta?*[bbox=11.54,48.14,11.543,48.145]')
YIELD value
UNWIND value["_children"] AS child
WITH child WHERE child["_type"] = "node"
WITH child.id AS id,
child.lat AS latitude,
child.lon AS longitude,
child["user"] AS userName
MERGE (point:Point {id: id})
SET point.latitude = latitude,
point.longitude = longitude
MERGE (user:User {name: userName})
MERGE (user)-[:EDITED]->(point)

76. UNWIND the array of elements
CALL apoc.load.xml('http://overpass.osm.rambler.ru/cgi/xapi_meta?*[bbox=11.54,48.14,11.543,48.145]')
YIELD value
UNWIND value["_children"] AS child
WITH child WHERE child["_type"] = "node"
WITH child.id AS id,
child.lat AS latitude,
child.lon AS longitude,
child["user"] AS userName
MERGE (point:Point {id: id})
SET point.latitude = latitude,
point.longitude = longitude
MERGE (user:User {name: userName})
MERGE (user)-[:EDITED]->(point)

77. Filter rows
CALL apoc.load.xml('http://overpass.osm.rambler.ru/cgi/xapi_meta?*[bbox=11.54,48.14,11.543,48.145]')
YIELD value
UNWIND value["_children"] AS child
WITH child WHERE child["_type"] = "node"
WITH child.id AS id,
child.lat AS latitude,
child.lon AS longitude,
child["user"] AS userName
MERGE (point:Point {id: id})
SET point.latitude = latitude,
point.longitude = longitude
MERGE (user:User {name: userName})
MERGE (user)-[:EDITED]->(point)

78. Apply Cypher transformation
CALL apoc.load.xml('http://overpass.osm.rambler.ru/cgi/xapi_meta?*[bbox=11.54,48.14,11.543,48.145]')
YIELD value
UNWIND value["_children"] AS child
WITH child WHERE child["_type"] = "node"
WITH child.id AS id,
child.lat AS latitude,
child.lon AS longitude,
child["user"] AS userName
MERGE (point:Point {id: id})
SET point.latitude = latitude,
point.longitude = longitude
MERGE (user:User {name: userName})
MERGE (user)-[:EDITED]->(point)
MERGE (point:Point {id: id})
SET point.latitude = latitude,
point.longitude = longitude
MERGE (user:User {name: userName})
MERGE (user)-[:EDITED]->(point)

79. Load JSON

80. apoc.load.json
WITH "https://api.stackexchange.com/2.2/questions?pagesize=100&order=desc&sort=creation&tagged=neo4j&site=stackoverflow&filter=!5-i6Zw8Y)4W7vpy91PMYsKM-k9yzEsSC1_Uxlf" AS url
CALL apoc.load.json(url) YIELD value
UNWIND value.items AS q
MERGE (question:Question {id:q.question_id})
ON CREATE SET question.title = q.title,
question.share_link = q.share_link,
question.favorite_count = q.favorite_count
MERGE (owner:User {id:q.owner.user_id})
ON CREATE SET owner.display_name = q.owner.display_name
MERGE (owner)-[:ASKED]->(question)
FOREACH (tagName IN q.tags |
MERGE (tag:Tag {name:tagName}) MERGE (question)-[:TAGGED]->(tag))
FOREACH (a IN q.answers |
MERGE (question)<-[:ANSWERS]-(answer:Answer {id:a.answer_id})
MERGE (answerer:User {id:a.owner.user_id})
ON CREATE SET answerer.display_name = a.owner.display_name
MERGE (answer)<-[:PROVIDED]-(answerer)
)

81. WITH "https://api.stackexchange.com/2.2/questions?pagesize=100&order=desc&sort=creation&tagged=neo4j&site=stackoverflow&filter=!5-i6Zw8Y)4W7vpy91PMYsKM-k9yzEsSC1_Uxlf" AS url
CALL apoc.load.json(url) YIELD value
UNWIND value.items AS q
MERGE (question:Question {id:q.question_id})
ON CREATE SET question.title = q.title,
question.share_link = q.share_link,
question.favorite_count = q.favorite_count
MERGE (owner:User {id:q.owner.user_id})
ON CREATE SET owner.display_name = q.owner.display_name
MERGE (owner)-[:ASKED]->(question)
FOREACH (tagName IN q.tags |
MERGE (tag:Tag {name:tagName}) MERGE (question)-[:TAGGED]->(tag))
FOREACH (a IN q.answers |
MERGE (question)<-[:ANSWERS]-(answer:Answer {id:a.answer_id})
MERGE (answerer:User {id:a.owner.user_id})
ON CREATE SET answerer.display_name = a.owner.display_name
MERGE (answer)<-[:PROVIDED]-(answerer)
)
Use FOREACH for arrays within a row
FOREACH (tagName IN q.tags |
MERGE (tag:Tag {name:tagName}) MERGE (question)-
[:TAGGED]->(tag))
FOREACH (a IN q.answers |
MERGE (question)<-[:ANSWERS]-(answer:Answer
{id:a.answer_id})
MERGE (answerer:User {id:a.owner.user_id})
ON CREATE SET answerer.display_name =
a.owner.display_name

82. Graph Refactorings

83. apoc.refactor.mergeNodes
MATCH (n:Person)
WITH n.email AS email, collect(n) as people
WHERE size(people) > 1
CALL apoc.refactor.mergeNodes(people)
YIELD node
RETURN node

84. apoc.refactor.mergeNodes
MATCH (n:Person)
WITH n.email AS email, collect(n) as people
WHERE size(people) > 1
CALL apoc.refactor.mergeNodes(people)
YIELD node
RETURN node

85. apoc.create.addLabels
MATCH (n:Movie)
CALL apoc.create.addLabels( id(n), [ n.genre ] ) YIELD node
REMOVE node.genre
RETURN node

86. Cypher Execution

87. Run large scale updates
CALL apoc.periodic.iterate(
'MATCH (n:Person) RETURN n',
'SET n.name = n.firstName + " " + n.lastName',
{batchSize:10000, parallel:true})

88. Graph Visualization
88

92. var viz;
function draw() {
var config = {
container_id: "viz",
server_url: "bolt://localhost:7687",
server_user: "neo4j",
server_password: "sorts-swims-burglaries",
labels: {
"Character": {
"caption": "name",
"size": "pagerank",
"community": "community"
}
},
relationships: {
"INTERACTS": {
"thickness": "weight",
"caption": false
}
},
initial_cypher: "MATCH (n)-[r:INTERACTS]->(m) RETURN *"
};
viz = new NeoVis.default(config);
viz.render();
}

93. var viz;
function draw() {
var config = {
container_id: "viz",
server_url: "bolt://localhost:7687",
server_user: "neo4j",
server_password: "sorts-swims-burglaries",
labels: {
"Character": {
"caption": "name",
"size": "pagerank",
"community": "community"
}
},
relationships: {
"INTERACTS": {
"thickness": "weight",
"caption": false
}
},
initial_cypher: "MATCH (n)-[r:INTERACTS]->(m) RETURN *"
};
viz = new NeoVis.default(config);
viz.render();
}

94. var viz;
function draw() {
var config = {
container_id: "viz",
server_url: "bolt://localhost:7687",
server_user: "neo4j",
server_password: "sorts-swims-burglaries",
labels: {
"Character": {
"caption": "name",
"size": "pagerank",
"community": "community"
}
},
relationships: {
"INTERACTS": {
"thickness": "weight",
"caption": false
}
},
initial_cypher: "MATCH (n)-[r:INTERACTS]->(m) RETURN *"
};
viz = new NeoVis.default(config);
viz.render();
}

95. GraphQL

96. GraphQL is a query language for your API, and a server-
side runtime for executing queries by using a type
system you define for your data.
What is it?

97. GraphQL is a query language for your API, and a
server-side runtime for executing queries by using a
type system you define for your data.
What is it?

98. GraphQL is a query language for your API, and a server-
side runtime for executing queries by using a type
system you define for your data.
What is it?

99. GraphQL is a query language for your API, and a server-
side runtime for executing queries by using a type
system you define for your data.
What is it?
type Planet {
name: String
climate: String
}
type Character {
name: String
friends: [Character]
homeWorld: Planet
species: Species
}
type Species {
name: String
lifespan: Int
origin: Planet
}

100. Manual mapping code

102. Auto translating GraphQL → Cypher
github.com/neo4j-graphql/neo4j-graphql
github.com/neo4j-graphql/neo4j-graphql-js

103. Server Side Extension

105. dbms.unmanaged_extension_classes=org.neo4j.graphql=/graphql
dbms.security.procedures.whitelist=graphql.*
conf/neo4j.conf

106. CALL graphql.idl('
type Movie {
title: String!
released: Int
actors: [Person] @relation(name:"ACTED_IN",direction:IN)
}
type Person {
name: String!
born: Int
movies: [Movie] @relation(name:"ACTED_IN")
}'
)

107. WITH '{ Person(born: 1961) { name, born } }' as query, {} as params
CALL graphql.execute(query,params)
YIELD result
UNWIND result.Person as p
RETURN p.name, p.born

108. JavaScript Library

110. The GRAND stack

111. The GRAND stack
GraphQL
React
Apollo
Neo4j Database

112. import {neo4jgraphql} from 'neo4j-graphql-js';
const resolvers = {
Query: {
Movie(object, params, ctx, resolveInfo) {
return neo4jgraphql(object, params, ctx, resolveInfo);
}
}
};

113. dzone.com/refcardz/an-overview-of-graphql

114. How do I find out
about more cool stuff?

116. This Week in Neo4j - Developer Update
neo4j.com/tag/twin4j

117. Get Dressed: Neo4j Graph Gear Store
graphgearstore.com

119. Thanks!