1. Kineograph: Taking the Pulse of a
Fast-Changing and Connected World
Speaker: LIN Qian
http://www.comp.nus.edu.sg/~linqian

2. Information
time-sensitive
rich connections

3. Challenges

4. 1. Timeliness guarantees

5. 2. Graph

6. 3. Graph-mining

7. Kineograph
distr. in-memory graph storage
incremental graph mining

8. Master Progress
Continuous table
Data feeds
Ingest
nodes Snapshooter
Graph
nodes Global consistent snapshots
Graph Storage
Computation
Incremental computation on a
static graph snapshot

9. Graph computation
Graph updates

10. Graph nodes
storage layer
computation layer

11. Storage layer
key/value store
logical partitions

12. Graph partitioning
edge-cut
no locality consideration

13. Snapshot
ingest nodes
graph nodes
global progress table

14. Ingest node
graph-update operations
sequence number

15. Epoch commit protocol

16. Progress table
s1 1
3
2
0
… … Global tx
vector
Ingest nodes s1 … sn sn 7
3
4
Snapshooter
Partition u Partition v
1 2 4 s1 2 3 5 s1
… Epoch specified by progress
…
Graph nodes
…
table and snapshooter
4 6 7 sn 5 6 8 sn

17. Graph update / compute
Pipeline
Incoming
Tweets … … Time
Snapshot Si-1 Si Si+1
Construction
Graph Epoch Ci
Computation ti-1 ti ti’ ti’’
Timeliness

18. Consistency
no global serialization
(diff. from 2PL or t.s. ordering)

19. Atomicity
v u
v u

20. Deterministic
vertex creation

21. Computation layer
incremental graph-mining

22. vertex-based
computation model

23. Incremental Graph
Computation
Updates from
other vertices
N
Detect Vertex Compute New Change
Init
Status Vertex Values Significantly?
Graph-Scale Propagate Y
Aggregation Updates

24. Push model
sender-side aggregation

25. Pull model
read a subset of neighbors

26. Execution model
BSP + Dynamic scheduling

27. 3 apps
TunkRank
SP
K-exposure

28. TunkRank

29. SP

30. K-exposure

31. Fault tolerance
among servers
Paxos-based solution

32. Ingest node failure
incarnation number

33. Fault tolerance
@ storage layer
quorum-based replication

34. Fault tolerance
@ computation layer
roll back & re-execute
primary/backup replication

35. Incremental expansion

36. Decaying

37. C#
17,000 LOC

38. Twitter feeds
8M vertices, 29M edges
100M tweets with 100K/sec
power-law

39. Graph-update throughput

40. Incremental vs.
Non-incremental

41. Scalability

42. Incoming data rate

43. Failure recovery