Graphs, what are they and why? Graph Data Management. Why do we need it? Problems in Distributed Graph How we solved the problems? Finding the value in Big Data.

1. www.Objectivity.com
Ibrahim Sallam
Director of Development

2. What we’ll talk about
• Graphs, what are they and why?
• Graph Data Management. Why do we need it?
• Problems in Distributed Graph
• How we solved the problems

3. Graphs

4. Graphs
http://opte.org
Simple Graph
Node <-> Node

5. The Value of Data
• “[finding] Japanese restaurants in New York
City liked by people from Japan” – a challenging
Facebook query.
• "The value of any piece of information is only
known when you can connect it with
something else that arrives at a future point in
time,” – CIA’s CTO Ira "Gus" Hunt

6. Connecting Data
Person Building
?
Work Live RR Visit Eat Shop

7. Relationships are everywhere
CRM, Sales &
Marketing Network
Mgmt, Teleco
m
Intelligence
(Government
& Business)
Finance
Healthcare
Research:
Genomics
Social
Networks
PLM (Product
Lifecycle
Mgmt)
IG

8. Graph Data
• Data structure for representing complicated
relationships between entities.
• Wide applications in…
– Bioinformatics.
– Social networks.
– Network management… etc.

9. Why a Graph Database ?

10. The Graph Specialist !
• Everyone specializes
– Doctors, Lawyers, Bankers, Developers 
• Why was data so normalized for so long !
• NoSQL is all about the data specialist
• Specializing in…
– Distribution / deployment
– Physical data storage
– Logical data model
– Query mechanism

11. What Problem!
Ingest
Update
Navigate
Massive graph data require efficient and intelligent tools
to analyze and understand it.

12. Scaling Writes
• Big/Fast data demands write performance
• Most NoSQL solutions allow you to scale
writes by…
– Partitioning the data
– Understanding your consistency requirements
– Allowing you to defer conflicts
Ingest

13. App-2
(Ingest V2)
App-2
(E23{ V2V3})
Scaling Graph Writes
ACID Transactions
InfiniteGraph
Objectivity/DB Persistence Layer
App-1
(Ingest V1)
App-3
(Ingest V3)
V1 V2 V3
App-1
(E1 2{ V1V2})
App-3
E12 E23
Ingest

14. High Performance Edge Ingest
IG Core/API
C1
C2
C3
E12
E23
TargetContainers
PipelineContainers
E(1->2)
E(3->1)
E(2->3)
E(2->1)
E(2->3)
E(3->1)
E(1->2)
E(3->2)
E(1->2)
E(2->3)
E(3->1)
E(2->1)
E(2->3)
E(3->1)
E(3->2)
E(1->2)
Pipeline
Agent
Ingest

15. Trade offs
• Excellent for efficient use of page cache
• Able to maintain full database consistency
• Achieves highest ingest rate in distributed
environments
• Almost always has highest “perceived” rate
• Trading Off :
• Eventual consistency in graph (connections)
• Updates are still atomic, isolated and durable but phased
• External agent performs graph building
Ingest

16. Result…
1 client
2 clients
4 clients
8 clients
0
50000
100000
150000
200000
250000
300000
350000
400000
450000
500000
1
2
4
NodesandEdgespersecond
1 client
2 clients
4 clients
8 clients
Ingest

17. Scaling Reads and Query
Distributed API
Application(s)
Partition 1 Partition 3Partition 2 Partition ...n
Processor Processor Processor Processor
Partitioning and Read Replicas… easy right !

18. Why are Graphs Different ?
Distributed API
Application(s)
Partition 1 Partition 3Partition 2 Partition ...n
Processor Processor Processor Processor
Navigate

19. Distributed Navigation
• Detect local hops and perform in memory
traversal
• Send the partial path to the distributed
processing to continue the navigation.
• Intelligently cache remote data when accessed
frequently
• Route tasks to other hosts when it is optimal
Navigate

20. Distributed Navigation Server
Processor
Distributed API
Partition 1 Partition 2
Processor
Application
A
X
Y
B
C
D
E
P(A,B,C,D)
F
G
Navigate

21. Schema – It’s not your enemy !
(at least not all the time...)
• Schema vs Schema-less
– Database religion
– InfiniteGraph supports schema, but does not
restrict connection types between vertices
– We take advantage of the schema when pruning.
– …Working on a hybrid support.

22. GraphViews
Leveraging Schema in the Graph
Patient Prescription
Drug
Ingredient
Outcome
Complaint
Visit
Allergy
Physician
Navigate

23. Schema Enables Views
• GraphViews are extremely powerful
• Allow Big Data to appear small !
• Connection inference can lead to exponential
gains in query performance
• Views are reusable between queries
• Built into the native kernel
Navigate

24. Why InfiniteGraph™?
• Objectivity/DB is a proven foundation
– Building distributed databases since 1993
– A complete database management system
• Concurrency, transactions, cache, schema, query, indexing
• It’s a Graph Specialist !
– Simple but powerful API tailored for data navigation.
– Easy to configure distribution model

25. Advanced Configured Placement
• Physically co-locate “closely related” data
• Driven through a declarative placement model
• Dramatically speeds “local” reads
Facility Data Page(s)Patient Data Page(s)
Mr
Citizen
Visit Visit
Dr
Jones
San
Jose
Facility
Dr
Smith
Primary
Physician
HasHas With
At
Located Located
Facility Data Page(s)
Dr
Blake
Sunny-
vale
Dr
Quinn
Located Located
With
At

26. Fully Distributed Data Model
Zone 2Zone 1
HostA
IG Core/API
Distributed Object and Relationship Persistence Layer
Customizable Placement
HostB HostC HostX
AddVertex()

27. Polyglot NoSQL Architectures
Distributed Data
Processing
Platform Document
Graph
Database
RDBMS
Partitioned Distributed DB (often Document / KV)
Users
Applications
External/LegacyData
TransformationMDM
Business

28. What else!
• Distributed update.
Update
… we are working on it.

29. Conclusion
Have we solved all the distributed graph problems?
Not quite, but…
We Learned to Stop Worrying.

30. QUESTIONS?