Kees Vegter, Field Engineer, Neo4j

1. Working With A Real World
Dataset In Neo4j
Kees Vegter
kees@neo4j.com
Modeling and Import

2. Kees Vegter!
Field Engineer @neo4j!
kees@neo4j.com!
kvegternu@neo4j-users.slack.com!

3. Agenda
• The Data!
• Build a graph data model
• Import
• LOAD CSV
• Import Tool (`neo4j-import`)
• Neo4j drivers
• Java

4. The Data!

5. hEps://www.yelp.com/dataset

6. Graph Data Model!

7. Labeled Property Graph Model

8. Labeled Property Graph Model

9. Our Data Model
hEp://www.apcjones.com/arrows/#
• IdenMfy “enMMes”
• What properMes are
relevant?
• IdenMfy unique ids
• Find connecMons
• Repeat

10. Yelp data set!

11. Cypher!

12. Cypher: Basic Structure
MATCH graph_pattern
RETURN results

13. graph_paKern: Nodes and RelaLonships
()-->()

14. graph_paKern: Labels and RelaLonship Types
(:Person)-[:FRIEND]->(:Person)

15. Cypher: Example Query
MATCH (p:Person {fullName :"Peter Giannetti"})-[r]-(n)
RETURN p, r, n
hEps://neo4j.com/docs/developer-manual/current/cypher/
hEps://www.opencypher.org/

16. Import!
LOAD CSV!

17. Create CSV files

18. What do the csv files look like?

19. Check first the csv contents
• The csv files are located in the Neo4j import Directory
• Check the csv contents witht the LOAD CSV Cypher command:
LOAD CSV WITH HEADERS FROM "file:///user.csv" AS row
return row limit 10
LOAD CSV WITH HEADERS FROM "file:///user.csv" AS row
MERGE (u:User {user_id: row.user_id})
SET u.name = row.name
, u.review_count = row.review_count
, u.average_stars = row.average_stars
, u.fans = row.fans;

20. Check first the csv contents
• The csv files are located in the Neo4j import Directory
• Check the csv contents witht the LOAD CSV Cypher command:
LOAD CSV WITH HEADERS FROM "file:///user.csv" AS row
return row limit 10
LOAD CSV WITH HEADERS FROM "file:///user.csv" AS row
MERGE (u:User {user_id: row.user_id})
SET u.name = row.name
, u.review_count = toInteger(row.review_count)
, u.average_stars = toFloat(row.average_stars)
, u.fans = toInteger(row.fans);

21. OpLmizing the load
• Inefficient cypher à use always 'EXPLAIN' to check and opMmize your
LOAD CSV statements.
• No indexes à import slows down when the dataset grows
• Create UNIQUE CONSTRAINTS, when using MERGE statements.
(Unique Constraints are backed by an Index)
• Too much data in one transacMon à OutOfMemoryExcepMon
• Use PERIODIC COMMIT to reduce the size of the transacMon
• Make sure the heap space for Neo4j is big enough (neo4j.conf)

22. EXPLAIN

23. How does Neo4j use indexes?
Indexes are only used to find the starting point
for queries.
Use index scans to look up
rows in tables and join them
with rows from other tables
Use indexes to find the starting
points for a query.
Relational
Graph

24. Create Index / Constraint
CREATE INDEX ON :Business(name);
CREATE CONSTRAINT ON (u:User) ASSERT u.user_id IS UNIQUE;
CREATE CONSTRAINT ON (b:Business) ASSERT b.business_id IS UNIQUE;
CREATE CONSTRAINT ON (c:Category) ASSERT c.name IS UNIQUE;
CREATE CONSTRAINT ON (b:Review) ASSERT b.review_id IS UNIQUE;
hEp://neo4j.com/docs/developer-manual/current/cypher/schema/constraints/
Constraint + index
Index

25. EXPLAIN

26. PERIODIC COMMIT
USING PERIODIC COMMIT 2000
LOAD CSV WITH HEADERS FROM “file:///review.csv" AS row
MATCH (u:User {user_id: row.user_id})
MATCH (b:Business {business_id: row.business_id})
MATCH (r:Review {review_id: row.review_id})
MERGE (u)-[:WROTE]->(r)
MERGE (r)-[:REVIEW_OF]->(b)
hEps://neo4j.com/docs/developer-manual/current/cypher/clauses/using-periodic-commit/

27. Naive Import
LOAD CSV WITH HEADERS FROM "file:///reviews.csv" AS row
MERGE (b:Business {business_id: row.business_id})
MERGE (u:User {user_id: row.user_id})
MERGE (r:Review {review_id: row.review_id})
ON CREATE SET r.stars = toInteger(row.stars),
r.text = row.text
MERGE (r)-[:REVIEW_OF]->(b)
MERGE (u)-[rr:WROTE]-(r)
ON CREATE SET rr.date = row.date

28. Break Up MERGEs
LOAD CSV WITH HEADERS FROM “file:///business.csv" AS row
MERGE (b:Business {business_id: row.business_id})
ON CREATE SET b.name = row.name …
LOAD CSV WITH HEADERS FROM “file:///user.csv” AS row
MERGE (b:User {user_id: row.user_id})
ON CREATE SET b.name = row.name …
LOAD CSV WITH HEADERS FROM “file:///review.csv" AS row
MATCH (u:User {user_id: row.user_id})
MATCH (b:Business {business_id: row.business_id})
MATCH (r:Review {review_id: row.review_id})
MERGE (u)-[:WROTE]->(r)
MERGE (r)-[:REVIEW_OF]->(b)
LOAD CSV WITH HEADERS FROM “file:///review.csv” AS row
MERGE (r:Review {review_id: row.review_id})
ON CREATE SET r.stars = row.stars …

29. cypher-shell
hEps://neo4j.com/docs/operaMons-manual/current/tools/cypher-shell/
cat simple_load_csv.cypher | …/bin/cypher-shell …
Replaces `neo4j-shell` in Neo4j 3.x+
Run multi-line Cypher scripts
--format=verbose (to get the same output as in 'neo4j-shell')
• Every cypher statement must be ended with a ';'
• It is possible to start, commit and rollback a transacMon
On Windows:
powershell
PS>type simple_load_csv.cypher | …bincypher-shell.bat …

30. Load csv cypher script example

31. Cypher: Calling procedures and funcLons
User defined FuncLons can be used in any expression or predicate, just
like built-in funcMons.
Procedures can be called stand-alone with CALL procedure.name();
But you can also integrate them into your Cypher statements which
makes them so much more powerful.
Load JSON example
WITH 'hEps://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/{branch}/src/test/resources/person.json' AS url
CALL apoc.load.json(url) YIELD value as person
MERGE (p:Person {name:person.name})
ON CREATE SET p.age = person.age, p.children = size(person.children)
hEps://github.com/neo4j-contrib/neo4j-apoc-procedures

32. Parallel Inserts w/ apoc.periodic.iterate
// Periodic iterate - LOAD CSV
WITH 'LOAD CSV WITH HEADERS
FROM "file:///business.csv"
AS row RETURN row' AS load_csv
CALL apoc.periodic.iterate(load_csv, '
MERGE (b:Business {business_id : row.business_id})
SET b.name = row.name'
, {batchSize: 5000, parallel: true
, iterateList: true, retries:3}) YIELD batches, total
RETURN *

33. Parallel Inserts w/ apoc.periodic.iterate
// Periodic iterate - LOAD CSV
WITH 'LOAD CSV WITH HEADERS
FROM "file:///business.csv"
AS row RETURN row' AS load_csv
CALL apoc.periodic.iterate(load_csv, '
MERGE (b:Business {business_id : row.business_id })
SET b.name = row.name'
, {batchSize: 5000, parallel: true
, iterateList: true, retries:3}) YIELD batches, total
RETURN *

34. Import!
apoc.load.*!

35. apoc.load.*
• There are mulMple apoc.load procedures you can load data from a lot of
external sources:
• json
• xml
• jdbc
• ...

36. apoc.load.jdbc
// Table check
with "jdbc:mysql://192.168.56.102:3306/yelp_db?user=yelp&password=yelp"
as url
call apoc.load.jdbc(url,"select * from user limit 10") yield row
return row
• No need for export to csv!
• Much more type save, no escaping of 'csv' breaking characters anymore.
• Just replace the LOAD CSV line with CALL apoc.load.jdbc(...) yield row

37. apoc.load.jdbc
// Import user table #user: 1183362
with "jdbc:mysql://192.168.56.102:3306/yelp_db?user=yelp&password=
as url
call apoc.load.jdbc(url,"select name, id as user_id
, review_count, average_stars, fans from user") yield row
MERGE (u:User {user_id: row.user_id})
SET u.name = row.name,
u.review_count = row.review_count,
u.average_stars = row.average_stars,
u.fans = row.fans
Can be placed in conf file!

38. apoc.load.jdbc
// Import friend table #friend: 39846890
with "jdbc:mysql://192.168.56.102:3306/yelp_db?user=yelp&password=
as url
call apoc.load.jdbc(url,"select friend_id, user_id from friend") y
MERGE (u:User {user_id: row.user_id} )
MERGE (f:User {user_id: row.friend_id} )
MERGE (u)-[:FRIENDS]->(f)

39. // Import business table #business: 156639
with "jdbc:mysql://192.168.56.102:3306/yelp_db?user=yelp&password=yelp"
as url
call apoc.load.jdbc(url,"select city, name, id as business_id
, stars, latitude, longitude, postal_code, address
, state, review_count, neighborhood from business") yield row
MERGE (b:Business {business_id: row.business_id})
SET b.address = row.address,
b.lat = row.latitude,
b.lon = row.longitude,
b.name = row.name,
b.city = row.city,
b.postal_code = row.postal_code,
b.state = row.state,
b.review_count = row.review_count,
b.stars = row.stars,
b.neighborhood = row.neighborhood

40. // Import (business) category #category: 590290
with "jdbc:mysql://192.168.56.102:3306/yelp_db?user=yelp&password=yelp"
as url
call apoc.load.jdbc(url,"select business_id, category from category") yield row
MATCH (b:Business {business_id: row.business_id})
MERGE (c:Category { name : row.category })
MERGE (b)-[:IN_CATEGORY]->(c)

41. // Import review table #review: 4736897
with "jdbc:mysql://192.168.56.102:3306/yelp_db?user=yelp&password=yelp"
as url
call apoc.load.jdbc(url,"select id as review_id, user_id
, business_id, text, stars, useful from review ") yield row
MERGE (b:Business {business_id: row.business_id})
MERGE (u:User {user_id: row.user_id})
MERGE (r:Review {review_id: row.review_id})
ON CREATE SET r.text = row.text,
r.date = row.date,
r.stars = row.stars,
r.useful = row.useful
MERGE (u)-[:WROTE]->(r)
MERGE (r)-[:REVIEWS]->(b)

42. load.jdbc cypher script example

43. Import!
Neo4j Import Tool!

44. Neo4j Import Tool (neo4j-import)
• Command line tool
• IniMal import only
• Creates foo.db
• Specific CSV file format
hEp://neo4j.com/docs/operaMons-manual/current/tutorial/import-tool/

46. neo4j-import file format
:ID(User) :LABEL name
123 User Will
124 User Bob
125 User Heather
126 User Erika
:ID(Review) :LABEL stars:int
127 Review 3
128 Review 2
129 Review 5
130 Review 1
:START_ID(User) :END_ID(Review) :TYPE
123 127 WROTE
124 128 WROTE
125 129 WROTE
126 130 WROTE
RelaMonships
Nodes

47. neo4j-import
neo4j-import --into /var/lib/neo4j/data/databases/yelp.db
--nodes user.csv
--nodes business.csv
--nodes review.csv
--relationships friends.csv
--relationships wrote.csv
--relationships review_of.csv
Now included in `neo4j-admin import`

48. neo4j-import advanced
neo4j-import --delimiter "," –stacktrace –into newdb
--nodes:User "userhdr.csv,n_user.csv"
--nodes:Business "businesshdr.csv,n_business.csv"
--nodes:Category "categoryhdr.csv,n_category.csv"
--nodes:Review "reviewhdr.csv,n_review.csv"
--relaMonships:FRIENDS "r_friendshdr.csv,r_friends.csv"
--relaMonships:IN_CATEGORY "r_in_categoryhdr.csv,r_in_category.csv"
--relaMonships:WROTE "r_wrotehdr.csv,r_wrote_review.csv"
--relaMonships:REVIEW_OF "r_review_oxdr.csv,r_review_of_business.csv"
Header outside the data
csv file! Label and RelaMonshiptype not
in the data (on every row)!

49. neo4j-import

50. neo4j-import
neo4j.conf Desktop App

51. Create Indexes
CREATE INDEX ON :User(name);
CREATE INDEX ON :Business(name);
CREATE INDEX ON :Review(stars);

52. Neo4j Drivers!

53. Neo4j Drivers
hEps://neo4j.com/developer/language-guides/

57. Import
methods!

58. • Fastest method (w.r.t. writes/second)
• IniMal import; a new database is being created
• Database is offline during import
• No need to create indexes in advance
• The cluster needs to be synchronized azer the import
58
$ ./bin/neo4j-import

59. 59
Import Approach

60. • May be the simplest method
• IniMal import or update
• Database is online during import, transacMonal!
• Create indexes upfront
• The cluster is being synchronized automaMcally
60
Cypher & LOAD CSV

61. 61
Cypher (LOAD CSV)

62. • Iterate / batching
• Plenty of procedures and funcMons
• GraphML
• JDBC
• .. and others (e.g. XML, JSON, …)
62
Cypher & APOC

63. • IniMal import or update
• Not transacMonal! Not thread-safe! Private API!
• But extremely fast
• Database is offline during import
• Can handle complex data transformaMons
• The cluster needs to be synchronized azer the import
63
BatchInserter

64. 64
BatchInserter

65. • Drivers for many languages available
• TransacMonal processing
• Batching
• ParallelizaMon possible
65
Driver via BOLT

66. 66
Driver via BOLT

67. So, which method should I
use?
67

68. It depends...
68
a bit
higher
effort
fast faster speed
$ ./neo4j-admin import
low
APOC
BatchInserter API
Driver/
BOLT
APOC
LOAD CSV

69. Querying The Graph!

70. The Graph

71. What Business Has Highest Reviews
MATCH (b:Business)<-[:REVIEW_OF]-(r:Review)
WITH b, avg(r.stars) AS mean
RETURN b.name, mean ORDER BY mean DESC LIMIT 25

72. hEp://www.lyonwj.com/scdemo/index.html

73. Refactoring!

74. Refactoring example: from city property to city
node
// mark the nodes you want to process first
apoc.periodic.iterate('match (b:Business) return b'
, 'SET b:InProcess'
,{batchSize: 1000, parallel : true, iterateList : true, retries: 3}) yield batches, total);
// create now a city node and a :LOCATED_IN relaMonship
// with apoc.periodic.commit
// this statement will be repeated unMl it returns 0
apoc.periodic.commit('match (b:InProcess)
with b limit $batchSize
MERGE (cit:City {name : b.city})
MERGE (b)-[:LOCATED_IN]->(cit)
REMOVE b.city
REMOVE b:InProcess
RETURN COUNT(b)' ,{batchSize: 10000});

75. Neo4j Sandbox
neo4jsandbox.com