What is Big Data and what is it not? When to use it and what's the added value? What to be aware of? What are the pitfalls?

1. big data
So What?
12 October 2016
1

2. Who am I?
• Software guy
• Technology leader with experience in software
development as CTOs and development managers
of mid-sized teams.
• Doing big data hands-on since 2009
• Running http://meetup.com/bigdatabe since 2011
(1700 members!)
2
@wimvanleuven
wim@bigboards.io

3. 3

4. 4

5. “Big data is data that exceeds the processing
capacity of conventional database systems.
The data is too big, moves too fast, or doesn’t fit
the strictures of your database architectures.”
5
–Edd Dumbill, O’Reilly
What is big data?
http://radar.oreilly.com/2012/01/what-is-big-data.html

6. …too big…
6
IOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOI
OIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOI
OOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOI
OIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIO
IIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOII
OIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIO

7. … moves to fast …
7

8. 8

9. … doesn’t fit …
9

10. What is Big Data not?
• not a delivery model (on-premise vs hosted vs
cloud vs IaaS/PaaS/SaaS vs serverless)
• not a deployment model (private, public, hybrid)
• not a revenue model (license vs subscription vs
Pay-as-you-Go)
• not software architecture
10

11. “We don’t do Hadoop because we have Big
Data; we do Big Data because we have
Hadoop.”
11
–Unknown developer, Facebook
What is Big Data? — revisited

12. New tools and technologies to capture and
process data on a cluster of commodity
hardware so that the system acts as one,
is resilient to failures and scales linearly.
12
What is Big Data? — revisited

13. Big Data is no panacea
13
• First decide what problem you want to solve; pick
a real business problem to add immediate value
• Start small, the technology is made for linear
scalability (a 3-node cluster is a cluster!)
• Then become lean: learn through experimentation

14. Big Data challenges
• Beware of hype, Big Data - washing and fad
• Tech infancy
• IT | Biz
• Data is hard
• Lack of skills!
14

15. Benefits
• Scalability of course
• Collect more and more data
• Robustness inherent to the setup
• More predictable performance
15

16. 16
Questions?

17. 17
Co-existence
Big
Data
View
ESB
App
ETL

18. DFS
18
1
2
3
4
5
2
4
5
1
2
5
1
3
4
2
3
5
1
3
4
Node A Node B Node C Node D Node E

19. MapReduce
19
4
5
3
2
1
Node A
Node B
Node C
Node D
Node E
Map Shuffle Reduce
x
y
z

20. 𝛌
20

21. 𝛋
21
3
1
2
45

22. 22
Q&A