Brief introduction to a new approach on handling big amount of data

1. HadoopDB Miguel Angel Pastor Olivar miguelinlas3 at gmail dot com http://miguelinlas3.blogspot.com http://twitter.com/miguelinlas3

3. HadoopDB Architecture

4. Results

5. Conclusions

6. Introduction

8. Data amount is exploding

9. Previous problem -> Shared nothing architectures

11. Map/Reduce systems

14. Analytics environments: not restart querys

15. Problem at scaling

18. UDF mechanism

19. Desirable SQL and no SQL interfaces

23. Assumption: failures are rare

24. Assumption: dozens of nodes in clusters

25. Engineering decisions

26. Background: Map/Reduce

28. Works on heterogeneus environment

31. SQL not supported directly ( Hive )

32. HadoopDB

39. Job and Task trackers

40. Architecture

43. Execute the SQL query

47. Plan to deploy as separated service

49. Breaking single data node in ckunks

53. Semantic analyzer connects to catalog

54. DAG of relational operators

55. Optimizer reestructuration

56. Convert plan to M/R jobs

57. DAG in M/R serialized in xml plan

60. Traverse DAG (bottom up). Rule based SQL generator

61. Benckmarking

64. 2 virtual cores

65. 850 GB storage

66. 64 bits Linux Fedora 8

68. 1024 MB heap size

70. PostgreSQL 8.2.5

71. No compress data

73. Used a cloud edition

75. Run on EC2 (not cloud edition available)

78. 18 millions ranking (~1Gigabyte)

79. Stored as plain text in HDFS

80. Loading data

81. Grep Task

85. UDF Aggregation Task

87. DBMS-X 15% overly optimistic

89. Fault tolerance and heterogeneus environments

90. Benchmarks

92. Reduce the number of nodes to achieve the same order of magnitude

93. Fault tolerance is important

94. Conclusions

96. PostgreSQL is not a column store

97. Hadoop and hive relatively new open source projects

98. HadoopDB is flexible and extensible

99. References

101. HadoopDB article

102. HadoopDB project

103. Vertica

104. Apache Hive

105. That´s all!