Slides of my Big Data talk at JAX2011 in Mainz, May 2011. For the demo code and detail description of the demo, a slide with URLs is included

1. Big Data Pavlo Baron

2. Pavlo Baron http://www.pbit.org [email_address] @pavlobaron

3. So, you think you’re having Big Data?

4. Don’t think

5. Know

6. Know your data

7. Know your data your scenarios

8. Know your data your scenarios how to scale

9. Know your data your scenarios how to scale the technology

10. Know your data your scenarios how to scale the technology when to stop

11. Know your data your scenarios how to scale the technology when to stop

12. Where does your data actually come from ?

13. Do you have a million well structured records?

14. Or a couple of Gigabytes of storage?

15. Does your data get modified every now and then ?

16. Do you look at your data once a month to create a management report?

17. Or is your data an unstructured chaos?

18. Do you get flooded by tera-/petabytes of data?

19. Or do you simply get bombed with data?

20. Does your data flow on streams at a very high rate from different locations?

21. Or do you have to read The Matrix ?

22. Do you need to distribute your data over the whole world

23. Or does your existence depend on (the quality of) your data?

24. Know your data your scenarios how to scale the technology when to stop

25. Is it the storage that you need to focus on?

26. Or are you more preparing data?

27. Or do you have your customers spread all over the world ?

28. Or do you have complex statistical analysis to do?

29. Or do you have to filter data as it comes?

30. Or is it necessary to visualize the data?

31. Know your data your scenarios how to scale the technology when to stop

32. To scale for Big Data means to...

33. Chop in smaller pieces

34. Chop in bite-size , manageable pieces

35. Separate reading from writing

36. Minimize hard relations

37. Separate archive from accessible data

38. Trash everything that has only to be analyzed in real-time

39. Parallelize and distribute

40. Strive after spatial proximity to processors

41. Utilize commodity hardware

42. Relax new hardware startup procedure

43. Consider hardware fallibility

44. Strive after spatial proximity to users

45. Consider network unreliability

46. Design with eventual actuality/consistency in mind

47. Design with Byzantine faults in mind

48. Consider latency an adjustment screw

49. Consider availability an adjustment screw

50. Be prepared for disaster

51. Utilize the fog/clouds

52. Design for theoretically unlimited amount of data

53. Design for frequent structure changes

54. Design for the all-in-one mix

55. Know your data your scenarios how to scale the technology when to stop

56. It’s not sufficient anymore just to throw it at your Oracle DB and to hope it works

57. You have no chance without science

58. To manage Big Data means to learn/know

59. To manage Big Data means to learn/know algorithms/ADTs

60. To manage Big Data means to learn/know algorithms/ADTs computing systems

61. To manage Big Data means to learn/know algorithms/ADTs computing systems networking

62. To manage Big Data means to learn/know algorithms/ADTs computing systems networking operating systems

63. To manage Big Data means to learn/know algorithms/ADTs computing systems networking operating systems database systems

64. To manage Big Data means to learn/know algorithms/ADTs computing systems networking operating systems database systems distributed systems

65. So, you know your data . You know your scenarios . You know the theory

66. Now pick the right tools for the job

67. I have thousands of log records per second . I want to store them immediately , but reliably for later statistics. How would I do that?

68. Consider DHTs , P2P systems, distributed data stores etc.

69. In order to write fast , distribute to several nodes with sloppy non-durable write quorum

70. Build upon a system implementing consistent hashing . Don’t try home-made sharding as distribution replacement - you will fail adding new nodes

71. Try Riak . It’s derived from Amazon Dynamo. It implements consistent hashing, gossip architecture, hinted handoff, vector clocks, merkle trees, sloppy quorum etc.

72. I need to analyze these records in real-time for patterns and send alerts when any of them match. How would I do that?

73. Consider CEP – complex event processing with an EPL – event processing language

74. Choose a sliding time window just big enough to recognize causality and fire events on thresholds, deviations, anomalies etc.

75. Try Esper . It implements CEP/EPL

76. I collect my data at different locations all over the world , but want to do statistical analysis in my headquarters or at one other location . How would I do that?

77. Aggregate your data and push it e.g. once a day out to the cloud . That’s a sort of replication if you like

78. Choose a cloud based data store which can store big objects . The store should provide similar consistency characteristics as your local data store

79. Try AWS (S3) or Rackspace (OpenStack/Swift) or a private cloud . They are either directly Dynamo based or implement similar concepts

80. That’s a lot of data and distribution. I need to quickly push it from a location into the cloud while data keeps coming in . How would I do that?

81. Use MapReduce to distribute the aggregation job to a group of nodes in order to quickly get the overall aggregation and cloud storage done

82. Map , sort , combine and reduce to whatever representation you need

83. Separate MapReduce splitting, jobs and intermediate storage from the local store to keep them independent and thus to read local store snapshots while still writing the new data

84. Try Hadoop . It implements MapReduce with an own file system (HDFS), distribution etc. It is highly extensible

85. I need to do some statistical analysis and visualize my data. How would I do that?

86. Choose a general purpose platform for statistical computing and graphics

87. Try R . It allows statistical analysis of whatever type of data and its graphical plotting. It’s highly extensible

88. And these are only some of the possible Q&As. There are more areas such as NoSQL , content preparation for CDN , data mining etc. which we didn’t consider

89. Know your data your scenarios how to scale the technology when to stop

90. The experiment – live demo Source code will be available on http://github.com/pavlobaron Detail description will be available on http://archi-jab-ture.blogspot.com

91. Situation Data Center (US) Data Center (EUR) Data Center (AFR) Votes Votes Reports

92. SoapUI Simulation HTTP server Esper Alert Riak Object Storage Hadoop R

93. We store votes as they come with sloppy write quorum. We store on several nodes in a regional cluster. We match patterns on a stream, not on saved data. We push aggregated day data from regions to the cloud using distributed MapReduce. We use scalable, distributed components with HA options. Etc. Does it scale ?

94. Thank you

95. Most images originate from istockphoto.com except few ones taken from Wikipedia and product pages