1. Giraph
bt22dr@gmail.com

2. Agenda
• Introduction
• BSP
• Pregel
• Giraph

3. Giraph

4. Giraph?

5. Apache Giraph

6. Apache Giraph is an iterative graph
processing system built for high
scalability.
- 출처 : http://giraph.apache.org/

7. - 출처 : http://giraph.apache.org/
Giraph originated as the open-
source counterpart to Pregel.

8. Pregel?

9. Google Pregel

10. Google Pregel is
distributed system especially developed
for large scale graph processing

11. 다양한 그래프 문제들
V E

12. 웹 그래프
소셜 네트워크
다양한 그래프 문제들
뉴스 기사의 유사성
질병 발생 경로
운송 경로
...

13. 인터넷
웹 그래프
뉴스 기사의 유사성
다양한 그래프 문제들
소셜 네트워크
질병 발생 경로
운송 경로
...
웹 2.0

14. 웹 그래프
소셜 네트워크
대규모 그래프
Internet of Things
모바일
인터넷
웹 2.0

15. 웹 그래프
소셜 네트워크
대규모 그래프
Internet of Things
모바일
billions of vertices, trillions of edges

16. Options?

17. Options?
• Crafting a custom distributed infrastructure

18. Options?
• Crafting a custom distributed infrastructure
• Relying on an existing distributed computing platform

19. Options?
• Crafting a custom distributed infrastructure
• Relying on an existing distributed computing platform
• Using a single-computer graph algorithm library

20. Options?
• Crafting a custom distributed infrastructure
• Relying on an existing distributed computing platform
• Using a single-computer graph algorithm library
• Using an existing parallel graph system

21. Options?
• Crafting a custom distributed infrastructure
• Relying on an existing distributed computing platform
• Using a single-computer graph algorithm library
• Using an existing parallel graph system
efficient processing of large graphs

22. Options?
locality of memory access
efficient processing of large graphs
• Crafting a custom distributed infrastructure
• Relying on an existing distributed computing platform
• Using a single-computer graph algorithm library
• Using an existing parallel graph system

23. Options?
locality of memory access
fault-tolerant platform
efficient processing of large graphs
• Crafting a custom distributed infrastructure
• Relying on an existing distributed computing platform
• Using a single-computer graph algorithm library
• Using an existing parallel graph system

24. Options?
locality of memory access
efficient processing of large graphs
• Crafting a custom distributed infrastructure
• Relying on an existing distributed computing platform
• Using a single-computer graph algorithm library
• Using an existing parallel graph system
general-purpose system
fault-tolerant platform

25. locality of memory access
efficient processing of large graphs
Pregel
general-purpose system
fault-tolerant platform
•
•
•
•

26. locality of memory access
efficient processing of large graphs
Pregel
general-purpose system
fault-tolerant platform
•
•
•
•
BSP

27. The Bulk Synchronous Parallel (BSP) abstract computer
is a bridging model for designing parallel algorithms
- 출처 : Bulk synchronous parallel - Wikipedia, the free encyclopedia

28. BSP computer :
- processors connected by a communication network
- fast local memory
- different threads of computation
- series of global supersteps

29. (series of) supersteps
… …
cf. MapReduce : (map / reduce) + (map / reduce) + (map / reduce) + …

30. superstep

31. superstep
독립적

32. superstep
독립적
단방향

33. superstep
독립적
단방향순서 고려 X

34. superstep
독립적
단방향
Costly
but attractive
순서 고려 X

36. V
user-define
function
S - 1
S + 1
superstep S

37. Pregel Computation
• Input : directed graph
• Sequence of supersteps
• output

38. Pregel Computation
• Input : directed graph
• Sequence of supersteps
• output
- Vertex ID
- Value
- Value (weight)
- Target vertex ID
- Value (weight)
- Target vertex ID
…-
V E E
- Vertex ID
- Value
- Value (weight)
- Target vertex ID
- Value (weight)
- Target vertex ID
…-
V E E
- Vertex ID
- Value
- Value (weight)
- Target vertex ID
- Value (weight)
- Target vertex ID
…-
V E E

39. Pregel Computation
• Input : directed graph
• Sequence of supersteps
• output
84

40. Pregel Computation
• Input : directed graph
• Sequence of supersteps
• output
84

41. Pregel Computation
• Input : directed graph
• Sequence of supersteps
• output
84
종료!!
…
4

42. Pregel Computation
• Input : directed graph
• Sequence of supersteps
• output
o the set of values explicitly output by the vertices
o aggregated statistics mined from the graph

43. Max value example

44. - 출처 : http://prezi.com/zghqtkqstrg-/apache-giraph-berlin-buzzwords/
MapReduce?

45. - 출처 : http://prezi.com/zghqtkqstrg-/apache-giraph-berlin-buzzwords/
MapReduce?

46. Pregel API

47. Pregel API
• Message Passing
• Combiners
• Aggregators
• Topology Mutations
• Input and Output

48. Master/Worker model
- 출처 : http://de.slideshare.net/sscdotopen/introducing-apache-giraph-for-large-scale-graph-processing

49. Master/Worker model
- 출처 : http://de.slideshare.net/sscdotopen/introducing-apache-giraph-for-large-scale-graph-processing

50. Fault Tolerance
• Checkpointing
o The master periodically instructs the workers to save the state of
their partitions to persistent storage
 e.g., Vertex values, edge values, incoming messages
• Failure detection
o Using regular “ping” messages
• Recovery
o The master reassigns graph partitions to the currently available
workers
o The workers all reload their partition state from most recent
available checkpoint

52. Giraph
• Open source implementation of Pregel
• Runs on Hadoop infrastructure
o map-only job in hadoop
• Computation is executed in memory
• Uses Apache ZooKeeper for synchronization
o If not exist, hadoop file system instead

53. Giraph
• Choose your graph generic types
o Vertex ID (type I)
o Vertex value (type V)
o Edge value (type E)
o Message value (type M)
• Define how to load the graph into Giraph
o Vertex Input Format
• Define how to store the graph from Giraph
o Vertex Output Format
• Override the compute() method

54. Giraph (Shortest Path example)
• generic types
• compute() method

55. Giraph (Shortest Path example)
• In/output format

56. Giraph (Shortest Path example)
1 2
3 4
1
3
1
2
10

57. 참고자료
• Pregel: A System for Large-Scale Graph Processing
• http://en.wikipedia.org/wiki/Bulk_synchronous_parallel
• http://giraph.apache.org
• http://prezi.com/zghqtkqstrg-/apache-giraph-berlin-buzzwords/