2. about me
who am i ...
➡ working as an it lead / software architect @ janssen pharmaceutica
• dealing with big scientific data sets
• hands-on expertise in big data and NoSQL technologies
➡ founder of datablend
• provide big data and NoSQL consultancy
Davy Suvee • share practical knowledge and big data use cases via blog
@DSUVEE
6. graphs ...
➡ graphs are continuously changing ...
➡ graphs and time ...
★ neo-versioning by david montag 1
2
★ representing time dependent graphs in neo4j by the isi foundation
★ modeling a multilevel index in neo4j by peter neubauer 3
copy and relink semantics
1. http://github.com/dmontag/neo4j-versioning 2. http://github.com/ccattuto/neo4j-dynagraph/wiki 3. http://blog.neo4j.org/2012/02/modeling-multilevel-index-in-neoj4.html
7. graphs ...
➡ graphs are continuously changing ...
➡ graphs and time ...
★ neo-versioning by david montag 1
2
★ representing time dependent graphs in neo4j by the isi foundation
★ modeling a multilevel index in neo4j by peter neubauer 3
copy and relink semantics
๏ graph size
๏ object identity
๏ mixing data-model and time-model
1. http://github.com/dmontag/neo4j-versioning 2. http://github.com/ccattuto/neo4j-dynagraph/wiki 3. http://blog.neo4j.org/2012/02/modeling-multilevel-index-in-neoj4.html
9. FluxGraph ...
➡ towards a time-aware graph ...
➡ implement a blueprints-compatible graph on top of Datomic
10. FluxGraph ...
➡ towards a time-aware graph ...
➡ implement a blueprints-compatible graph on top of Datomic
➡ make FluxGraph fully time-aware
★ travel your graph through time
★ time-scoped iteration of vertices and edges
★ temporal graph comparison
12. travel through time
FluxGraph fg = new FluxGraph();
Davy
Vertex davy = fg.addVertex();
davy.setProperty(“name”,”Davy”);
13. travel through time
FluxGraph fg = new FluxGraph();
Davy
Vertex davy = fg.addVertex();
davy.setProperty(“name”,”Davy”);
Peter
Vertex peter = ...
14. travel through time
FluxGraph fg = new FluxGraph();
Davy
Vertex davy = fg.addVertex();
davy.setProperty(“name”,”Davy”);
Peter
Vertex peter = ...
Vertex michael = ...
Michael
15. travel through time
FluxGraph fg = new FluxGraph();
Davy
kn
ow
Vertex davy = fg.addVertex();
s
davy.setProperty(“name”,”Davy”);
Peter
Vertex peter = ...
Vertex michael = ...
Edge e1 = Michael
fg.addEdge(davy, peter,“knows”);
16. travel through time
Davy
Date checkpoint = new Date();
kn
ow
s
Peter
Michael
17. travel through time
Davy
Date checkpoint = new Date();
kn
ow
s
davy.setProperty(“name”,”David”); Peter
Michael
18. travel through time
David
Date checkpoint = new Date();
kn
ow
s
davy.setProperty(“name”,”David”); Peter
Michael
19. travel through time
David
Date checkpoint = new Date();
kn
ow
s
davy.setProperty(“name”,”David”); Peter
kn
Edge e2 =
ow
fg.addEdge(davy, michael,“knows”);
s
Michael
20. travel through time by default
time
kn
Davy ow David
Davy
s
kn
ow
checkpoint
s
current
Peter Peter
kn
ow
s
Michael Michael
21. travel through time
time
kn
Davy ow David
Davy
s
kn
ow
checkpoint
s
current
Peter Peter
kn
ow
s
Michael Michael
fg.setCheckpointTime(checkpoint);
22. time-scoped iteration
t1 t2 t3 tcurrrent
change change change
Davy Davy’ Davy’’ Davy’’’
➡ how to find the version of the vertex you are interested in?
23. time-scoped iteration
t1 t2 t3 tcurrrent
next next next
Davy Davy’ Davy’’ Davy’’’
previous previous previous
Vertex previousDavy = davy.getPreviousVersion();
24. time-scoped iteration
t1 t2 t3 tcurrrent
next next next
Davy Davy’ Davy’’ Davy’’’
previous previous previous
Vertex previousDavy = davy.getPreviousVersion();
Iterable<Vertex> allDavy = davy.getNextVersions();
25. time-scoped iteration
t1 t2 t3 tcurrrent
next next next
Davy Davy’ Davy’’ Davy’’’
previous previous previous
Vertex previousDavy = davy.getPreviousVersion();
Iterable<Vertex> allDavy = davy.getNextVersions();
Iterable<Vertex> selDavy = davy.getPreviousVersions(filter);
26. time-scoped iteration
t1 t2 t3 tcurrrent
next next next
Davy Davy’ Davy’’ Davy’’’
previous previous previous
Vertex previousDavy = davy.getPreviousVersion();
Iterable<Vertex> allDavy = davy.getNextVersions();
Iterable<Vertex> selDavy = davy.getPreviousVersions(filter);
Interval valid = davy.getTimerInterval();
28. time-scoped iteration
➡ When does an element change?
➡ vertex:
★ setting or removing a property
★ add or remove it from an edge
★ being removed
29. time-scoped iteration
➡ When does an element change?
➡ vertex: ➡ edge:
★ setting or removing a property ★ setting or removing a property
★ add or remove it from an edge ★ being removed
★ being removed
30. time-scoped iteration
➡ When does an element change?
➡ vertex: ➡ edge:
★ setting or removing a property ★ setting or removing a property
★ add or remove it from an edge ★ being removed
★ being removed
➡ ... and each element is time-scoped!
36. use case: longitudinal patient data
t1 t2 t3 t4 t5
smoking smoking death
patient patient patient patient patient
cancer cancer
37. use case: longitudinal patient data
➡ historical data for 15.000 patients over a period of 10 years (2001- 2010)
38. use case: longitudinal patient data
➡ historical data for 15.000 patients over a period of 10 years (2001- 2010)
➡ example analysis:
★ if a male patient is no longer smoking in 2005
★ what are the chances of getting lung cancer in 2010, comparing
patients that smoked before 2005
patients that never smoked
40. use case: longitudinal patient data
➡ get all male non-smokers in 2005
fg.setCheckpointTime(new DateTime(2005,12,31).toDate());
41. use case: longitudinal patient data
➡ get all male non-smokers in 2005
fg.setCheckpointTime(new DateTime(2005,12,31).toDate());
Iterator<Vertex> males =
fg.getVertices("gender", "male").iterator()
42. use case: longitudinal patient data
➡ get all male non-smokers in 2005
fg.setCheckpointTime(new DateTime(2005,12,31).toDate());
Iterator<Vertex> males =
fg.getVertices("gender", "male").iterator()
while (males.hasNext()) {
Vertex p2005 = males.next();
boolean smoking2005 =
p2005.getEdges(OUT,"smokingStatus").iterator().hasNext();
}
44. use case: longitudinal patient data
➡ which patients were smoking before 2005?
boolean smokingBefore2005 =
((FluxVertex)p2005).getPreviousVersions(new TimeAwareFilter() {
public TimeAwareElement filter(TimeAwareElement element) {
return element.getEdges(OUT, "smokingStatus").iterator().hasNext()
? element : null;
}
}).iterator();
46. use case: longitudinal patient data
➡ which patients have cancer in 2010
Graph g =
fg.difference(smokerws,
time2010.toDate(),
time2005.toDate());
47. use case: longitudinal patient data
➡ which patients have cancer in 2010
working set of smokers
Graph g =
fg.difference(smokerws,
time2010.toDate(),
time2005.toDate());
48. use case: longitudinal patient data
➡ which patients have cancer in 2010
working set of smokers
Graph g =
fg.difference(smokerws,
time2010.toDate(),
time2005.toDate());
➡ extract the patients that have an edge to the cancer node
52. gephi plugin for blueprints!
1
➡ available on github
http://github.com/datablend/gephi-blueprints-plugin
➡ Support for neo4j, orientdb, dex, rexter, ...
1. Kudos to Timmy Storms (@timmystorms)
