How to use Neo4j in an Elixir project with real time updates. Presented September 8, 2017 at ElixirConf 2017.

1. Elixir + Neo4j
Regina Imhoff

2. Regina Imhoff
StabbyMcDuck
StabbyMcDuck
BlueRosebud
Project: github.com/StabbyMcDuck/elixir_ravelry

3. What is Neo4j?
• A graph database
• NoSQL / non-relational
• Uses Cypher for queries
• ACID compliant
• Just like many relational databases!

4. What is Neo4j?
• CRUD
• Connected data
• Social networks
• Graph-based search
• Real-Time Recommendations
• More!

5. What is a graph?

6. Charles
Annie Betty
Follows
Follow
s Follow
s
Follows

7. Node
(Vertex)
Charles
Annie Betty
Follows
Follow
s Follow
s
Follows

8. Node
(Vertex)
Relationship
(Edge)
Charles
Annie Betty
Follows
Follow
s Follow
s
Follows

9. Node
(Vertex)
Relationship
(Edge)
Two directed relationships
Charles
Annie Betty
Follows
Follow
s Follow
s
Follows

10. Labeled Property Graph Model
• Nodes
• Contain properties (key-value pairs)
• Have one or more labels
• Relationships
• Have names
• Are directed
• Can also contain properties

11. Why Use a Graph Database?
• Performance
• SQL performance gets worse with large data sets
• Especially compared to join-intensive queries in SQL
• Graph databases remain mostly constant
• Queries are localized to a section the graph

12. Why Use a Graph Database?
• Flexibility
• Graph databases are additive
• Can add new kinds of relationships, nodes, labels,
subgraphs, etc. to existing structure
• Won't disturb existing queries!
• Less developer time spent on modeling domains

13. Why Use a Graph Database?
• Agility
• Schema-free
• Change the data model as you develop

14. Elixir Ravelry Project
• Replicating a portion of Ravelry.com
• The manufacturing and sales of yarn
• Applicable to most social and manufacturing
processes

15. Elixir Ravelry
BFL image: http://www.ansi.okstate.edu/breeds/sheep/bluefacedleicester/

16. Wool
Cards
Material For
User
Roving
Carding image: http://westernreservepublicmedia.org/halefarm/loghouse_wool.htm

17. Owns
User
Owns
User
Wool
Cards
Material For
User
Roving

18. Roving
Material For Dyed
Roving
User
Dyes
Roving image: https://www.etsy.com/shop/Pigeonroof

19. Dyed
Roving
Spins
Material For
Yarn
User

20. Plies image: http://www.gangalibas.com/
Dyed
Roving
Spins
Material For
Yarn
User

21. Dyed
Roving
Spins
Material For
Material For
Yarn
User
Roving
Material For

22. Material For
Knits
Project
User
Yarn
Patterns
Authors
Pattern
User

23. Wool
Roving
User
Cards
Material For
Dyed
Roving
Material For
User
Dyes
Yarn
Material For
User
Spins
Material For
Project
Material For
User
Knits
Pattern
Patterns
User
Authors
Material For

24. 😱☠🐑 RUT-ROH! 🐑☠😱
• Scenario: somebody who purchased a skein of yarn
has tested positive for anthrax!
• We need to find all the people who were involved in
the production of this dye lot to test them too

25. SQL Query
// Spinner
SELECT users.*
FROM users
INNER JOIN spinnings
ON spinnings.spinner_id = users.id
INNER JOIN yarn
ON yarn.material_id = spinnings.id
INNER JOIN users as knitters
ON knitters.id = yarn.knitter_id
WHERE knitters.name = "Bob"
UNION

26. SQL Query
// Dyed Roving
SELECT users.*
FROM users
INNER JOIN dyed_roving
ON dyed_roving.dyer_id = users.id
INNER JOIN spinnings
ON spinnings.material_id = dyed_roving.id
INNER JOIN yarn
ON yarn.material_id = spinnings.id
INNER JOIN users as knitters
ON knitters.id = yarn.knitter_id
WHERE knitters.name = "Bob"
UNION

27. SQL Query
// Roving
SELECT users.*
FROM users
INNER JOIN roving
ON roving.carder_id = users.id
INNER JOIN dyed_roving
ON dyed_roving.material_id = roving.id
INNER JOIN spinnings
ON spinnings.material_id = dyed_roving.id
INNER JOIN yarn
ON yarn.material_id = spinnings.id
INNER JOIN users as knitters
ON knitters.id = yarn.knitter_id
WHERE knitters.name = "Bob"

28. Cypher Query
MATCH (p:User)-[:OWNS]->(y:Yarn)-[*]-(q:User)
WHERE p.name = ‘Bob'
AND y.name = ‘sherlock'
RETURN q

29. Cypher Query
MATCH (p:User)-[:OWNS]->(y:Yarn)-[*]-(q:User)
WHERE p.name = ‘Bob'
AND y.name = ‘sherlock'
RETURN q
Variable

30. Cypher Query
MATCH (p:User)-[:OWNS]->(y:Yarn)-[*]-(q:User)
WHERE p.name = ‘Bob'
AND y.name = ‘sherlock'
RETURN q
Variable
Node

31. Cypher Query
MATCH (p:User)-[:OWNS]->(y:Yarn)-[*]-(q:User)
WHERE p.name = ‘Bob'
AND y.name = ‘sherlock'
RETURN q
Variable
Relationship
Node

32. Cypher Query
MATCH (p:User)-[:OWNS]->(y:Yarn)-[*]-(q:User)
WHERE p.name = ‘Bob'
AND y.name = ‘sherlock'
RETURN q
Variable
Relationship Variable length pathNode

33. Cypher Query
MATCH (p:User)-[:OWNS]->(y:Yarn)-[*]-(q:User)
WHERE p.name = ‘Bob'
AND y.name = ‘sherlock'
RETURN q
Variable
Relationship Variable length path TypeNode

34. Cypher Query
MATCH (p:User)-[:OWNS]->(y:Yarn)-[*]-(q:User)
WHERE p.name = ‘Bob'
AND y.name = ‘sherlock'
RETURN q
Variable
Relationship
Directed Relationship
Variable length path TypeNode

35. Cypher
• Neo4j local browser
• Make queries
• See data
• Interact with data

37. Enter Bolt.Sips
• hexdocs.pm/bolt_sips
• Neo4j driver for Elixir
• Uses Bolt protocol
• Supports Cypher queries
• Connection pool

38. What about Ecto?
• Ecto Query
• Not for any graph databases ☹
• Ecto Repo
• No adapter for Neo4j 😢
• Ecto Schema
• No nodes, no relationships 😭

39. Manage Transactions
setup %{conn: conn} do
bolt_sips_conn =
Bolt.Sips.conn()
!|> Bolt.Sips.begin()
on_exit fn !->
Bolt.Sips.rollback(bolt_sips_conn)
end
conn = Plug.Conn.put_private(conn, :bolt_sips_conn, bolt_sips_conn)
%{bolt_sips_conn: bolt_sips_conn, conn: conn}
end

40. Manage Transactions
setup %{conn: conn} do
bolt_sips_conn =
Bolt.Sips.conn()
!|> Bolt.Sips.begin()
on_exit fn !->
Bolt.Sips.rollback(bolt_sips_conn)
end
conn = Plug.Conn.put_private(conn, :bolt_sips_conn, bolt_sips_conn)
%{bolt_sips_conn: bolt_sips_conn, conn: conn}
end

41. Manage Transactions
setup %{conn: conn} do
bolt_sips_conn =
Bolt.Sips.conn()
!|> Bolt.Sips.begin()
on_exit fn !->
Bolt.Sips.rollback(bolt_sips_conn)
end
conn = Plug.Conn.put_private(conn, :bolt_sips_conn, bolt_sips_conn)
%{bolt_sips_conn: bolt_sips_conn, conn: conn}
end

42. bolt_sips_conn
def bolt_sips_conn(conn) do
Map.get_lazy(conn.private,
:bolt_sips_conn,
&Bolt.Sips.conn/0)
end
def index(conn, _params) do
dyed_roving = conn
!|> bolt_sips_conn()
!|> Repo.DyedRoving.list()
json conn, dyed_roving
end

43. defmacro !__using!__([]) do
quote do
alias ElixirRavelry.Repo
def get(conn, id) do
Repo.get_node(conn, type(), id)
end
def graph(conn, id, direction) do
Repo.graph(conn, type(), id, direction)
end
def list(conn) do
Repo.list_node(conn, type())
end
end
end

44. @callback row_to_struct(%Bolt.Sips.Types.Node{}) !::
struct
@callback create(conn !:: %Bolt.Sips.Connection{}, struct) !::
struct

45. def create(conn, %Owns{started_at: started_at, user_id: user_id, wool_id: wool_id}) do
Repo.create_relationship(conn,
%{type: type(),
end_node_id: wool_id,
start_node_id: user_id,
started_at: Repo.to_timestamp(started_at)})
end

46. def row_to_struct(
%Bolt.Sips.Types.Relationship{
"end": wool_id,
id: id,
properties: %{
"started_at" !=> started_at_timestamp
},
start: user_id,
type: type()
}
) do
%Owns{
!__meta!__: %Ecto.Schema.Metadata{
source: {nil, type()},
state: :loaded
},
id: id,
wool_id: wool_id,
user_id: user_id,
started_at: Repo.from_timestamp(started_at_timestamp)
}
end

47. def row_to_struct(
%Bolt.Sips.Types.Relationship{
"end": wool_id,
id: id,
properties: %{
"started_at" !=> started_at_timestamp
},
start: user_id,
type: type()
}
) do
%Owns{
!__meta!__: %Ecto.Schema.Metadata{
source: {nil, type()},
state: :loaded
},
id: id,
wool_id: wool_id,
user_id: user_id,
started_at: Repo.from_timestamp(started_at_timestamp)
}
end

48. def row_to_struct(
%Bolt.Sips.Types.Relationship{
"end": wool_id,
id: id,
properties: %{
"started_at" !=> started_at_timestamp
},
start: user_id,
type: type()
}
) do
%Owns{
!__meta!__: %Ecto.Schema.Metadata{
source: {nil, type()},
state: :loaded
},
id: id,
wool_id: wool_id,
user_id: user_id,
started_at: Repo.from_timestamp(started_at_timestamp)
}
end

49. def graph(conn, type ! nil, id, direction, options) do
n_type = if type do
":!#{type}"
else
""
end
conn
!|> Bolt.Sips.query!(
"""
MATCH (n!#{n_type})
WHERE id(n) = toInteger({id})
!#{graph_optional_match(direction, options)}
WITH !#{graph_with(direction)}
RETURN !#{graph_return(direction)}
""",
cypher_parameters(id, options)
)
!|> graph_return_to_map()
end

50. defp graph_optional_match("forward", options) do
"OPTIONAL MATCH forward = (n)-[forward_relationship*0!..]!->(sink!#{forward_type(options)})"
end
defp graph_optional_match("both", options) do
"!#{graph_optional_match("forward", options)}n!#{graph_optional_match("backwards", options)}"
end
defp graph_optional_match("backwards", options) do
"OPTIONAL MATCH backwards = (source!#{backwards_type(options)})-[backwards_relationship*0!..]!->(n)"
end

51. defp graph_optional_match("forward", options) do
"OPTIONAL MATCH forward = (n)-[forward_relationship*0!..]!->(sink!#{forward_type(options)})"
end

52. conn = get conn,
"/api/v1/projects/!#{project.id}/graph",
%{"direction" !=> "backwards", "type" !=> "User"}

53. MATCH (n#{n_type})
WHERE id(n) = toInteger({id})
OPTIONAL MATCH backwards = (source#{backwards_type(options)})-
[backwards_relationship*0..]->(n)
WITH COLLECT(DISTINCT source) as source_nodes,
COLLECT(DISTINCT head(backwards_relationship)) as backwards_rels
RETURN source_nodes, backwards_rels

54. MATCH (n#{n_type})
WHERE id(n) = toInteger({id})
OPTIONAL MATCH backwards = (source#{backwards_type(options)})-
[backwards_relationship*0..]->(n)
WITH COLLECT(DISTINCT source) as source_nodes,
COLLECT(DISTINCT head(backwards_relationship)) as backwards_rels
RETURN source_nodes, backwards_rels

55. Real Time Updates
• The Goal:
• See changes to the graph as it is updated
• The Problem:
• Don’t want to send the whole graph every single time it’s updated
• The Solution:
• Store the previously sent graph on the socket

56. Real Time Updates
• Nodes are created before relationships
• Relationships need a start node and an end node
• Monitor when relationships are created

57. Dyed
Roving

58. Dyed
Roving Yarn

59. Dyed
Roving
Material For
Yarn

60. intercept ["graph_update"]
def handle_out(event = "graph_update",
%{nodes: nodes, relationships: relationships},
socket = %Phoenix.Socket{
assigns: %{node_id_set: node_id_set,
relationship_id_set: relationship_id_set}}) do
# !!...Calculate filtered graph!!...
push socket, event, filtered_graph
new_socket = socket
!|> assign(:node_id_set, new_node_id_set)
!|> assign(:relationship_id_set,
new_relationship_id_set)
{:noreply, new_socket}
end

61. Real Time Updates
1. Create relationship in Repo
2. Graph query on start or end node id of relationship
3. Broadcast for each node in graph to that node’s graph id topic

62. Testing Real Time Updates
conn = post conn, "/api/v1/material-for",
%{start_node_id: yarn.id, end_node_id: project.id}
conn =
conn
!|> recycle()
!|> Plug.Conn.put_private(:bolt_sips_conn, bolt_sips_conn)
!|> post("/api/v1/cards", %{user_id: cards_user.id,
roving_id: roving.id})

63. topic = “graph:!#{project.id}"
assert_receive %Phoenix.Socket.Message{
event: "graph_update",
topic: ^topic,
payload: %{
nodes: first_nodes,
relationships: _first_relationships
}
}

64. Future Work
• Ecto Adapter
• Use Ecto Schema to do type conversions
• Remove need to pass Bolt.Sips.Conn everywhere
• Query Library
• Ecto Query won’t support Cypher keywords

65. Additional Resources
• O’Reilly Graph Databases
• Free download: neo4j.com/graph-databases-book
• Cypher Refcard
• neo4j.com/docs/cypher-refcard/current
• GraphConnect
• graphconnect.com

66. 💖💖💖
• Florin Patrascu @florin
• Karin Wolok @hellojewfro
• Luke Imhoff @kronicdeth