1. Big Data, Disruption
and the 800 Pound
Gorilla in the Corner
Michael Stonebraker

2. Big Data, Disruption and the 800 Pound Gorilla in the Corner 2
Speakers
Michael Stonebraker
Adjunct Professor at MIT and
Tamr Co-founder
Mingo Sanchez
Sales Engineer

3. Big Data, Disruption and the 800 Pound Gorilla in the Corner
The Meaning of Big Data - 3 V’s
• Big Volume
— simple (SQL) analytics
— complex (non-SQL) analytics
• Big Velocity
— Drink from a fire hose
• Big Variety
— Large number of diverse data sources to integrate
3

4. Big Data, Disruption and the 800 Pound Gorilla in the Corner 4
Big Volume - Little Analytics
• Well addressed by data warehouse crowd
• Who are pretty good at SQL analytics on
• Hundreds of nodes
• Petabytes of data

5. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Big Volume - Little Analytics
• Architecture
• Multi-node, partitioned, parallel column stores
• Except Oracle!!!
• No gorilla here
• But two disrupters are looming
5

6. First Disrupter– The Cloud
• Everybody will move there sooner or later
• Hamilton’s vignette
• Dewitt’s vignette
• You should be planning on moving decision support to
the cloud aggressively
• And cloud DBMSs are all moving to support elasticity
• Apply 1000 nodes or 5, depending on query requirements – pay
for what you need
6

7. Big Data, Disruption and the 800 Pound Gorilla in the Corner
First Disrupter– The Cloud
● AWS plays by different rules
• S3 has a dramatic (artificial) pricing advantage
relative to EBS
• House solutions are favored
• >50 tee shirt sizes!!!!
● Cloud architecture is a challenge!
7

8. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Second Disrupter — Warehouses are
Yesterday’s Issue
• Data science will supersede business
intelligence
• As soon as enterprises can hire enough
competent data scientists
• After all would you like a predictive model or
a big table of numbers
8

9. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Second Disrupter — Warehouses are
Yesterday’s Issue
● Data Science is machine learning
• Deep learning
• Conventional machine learning (Deep learning will not take
over the world because of training data and explainability)
● Data Science is Non-SQL data analytics (PCA, SVD, …)
● Whatever your marketing department comes up with!
9

10. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Data Science is
Big Data - Big Analytics
• Complex math operations (machine learning, clustering,
trend detection, ….)
• The world of the “quants” and the “rocket scientists”
• Mostly specified as linear algebra on array data
• A dozen or so common ‘inner loops’
• Matrix multiply
• QR decomposition
• SVD decomposition
• Linear regression
• Run on CPUs, GPUs and/or other stuff
10

11. Solution Options
ML Packages
• SciKit-learn, tensor flow, …
• Code for popular algos (SciKit)
• Platform to code custom algo (tensor flow)
• No data management or persistence!
11

12. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Solution Options
R, SAS, Matlab, et al
● Weak or non-existent data management
● File system storage
● R doesn’t scale and is not a parallel system
12

13. Solution Options
RDBMS alone
● Code analytics in SQL
• Orders of magnitude too slow!!
• Tamr quickly discarded this tactic
● Coding operations as UDFs still requires you to
simulate arrays on top of tables
• And current UDF model not powerful enough to support
iteration
• Have to code in the UDF
13

14. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Array DBMSs
● SciDB, TileDB, et. al.
• Storage and filtering
• With competitive built-in algo
• Traction in genomics space
14

15. Solution Options
Spark
● Spark SQL is not competitive
● Neither is Spark Streaming
● Spark is popular parallel platform – do your own
plumbing
• Tamr uses Spark, HBase, Elastic search,
Postgres
15

16. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Solution Options
Map Reduce / Hadoop
● Not good for anything
• Map Reduce discarded by Google 5 years ago
● Hadoop has been rebranded as HDFS plus a bunch of
other stuff
● Remember that the cloud vendors have more
complete offerings (S3 plus …) in this area.
16

17. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Big Data — Big Analytics
● The wild west right now
• Hold onto your seat belt!
17

18. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Big Velocity
• Sensor tagging everything of value sends velocity
through the roof
• E.g. car insurance
• Smartphones as a mobile platform sends velocity
through the roof
• State of multiplayer internet games must be
recorded — sends velocity through the roof
18

19. Big Data, Disruption and the 800 Pound Gorilla in the Corner
• Big pattern — little state (electronic trading)
• Find me a ‘strawberry’ followed within 100 ms by a ‘banana’
• Complex event processing (CEP) is focused on this
problem
• Patterns in a firehose
P.S. I started StreamBase but I have no current relationship with the
company
Two Different Solutions
19

20. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Big Pattern – Little State
● CEP is pretty mature
• Offerings from Tibco, Kafka, …
● What transaction model do you want?
• Exactly once semantics for sure
• Failure model?
20

21. Big Data, Disruption and the 800 Pound Gorilla in the Corner
• Big state - little pattern
• For every security, assemble my real-time global position
• And alert me if my exposure is greater than X
• Looks like high performance OLTP
• Want to update a database at very high speed
21
Two Different Solutions

22. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Solution Choices for New OLTP
Old SQL
● The elephants
• Slooooow!!!!
• See “Through the OLTP Looking Glass and
What We Found There” VLDB 2007
22

23. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Solution Choices for New OLTP
No SQL
● Give up SQL
• Interesting to note that Cassandra
and Mongo are moving to (yup) SQL
● Give up ACID
• If you need ACID, this is a decision to
tear your hair out by doing it in user
code
• Can you guarantee you won’t need
ACID tomorrow?
23

24. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Solution Choices for New OLTP
New SQL
● Keep ACID and SQL, but
• Use main memory
• Different XACT implementation
● Products from Microsoft, IBM, VoltDB, MemSQL, …
● I have not seen an application that overruns CEP
or NewSQL
• No gorilla here
24

25. Big Variety – The 800 Pound Gorilla
• Scenario #1 – Data Scientists
• Scenario #2 – Integration of silos
25

26. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Data Science
• Data scientist has an idea – e.g. does Rogaine cause weight
gain in mice?
• He must
• Find relevant data sets (Merck has 4000 or so Oracle data bases plus a
big data lake plus files) – and the public web is a treasure chest of info
• Perform data integration on the results
• A quote from a data scientist at iRobot “I spend 90% of my
time finding and cleaning data and then 90% of the other
10% checking the cleaning”
26

27. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Enterprise Problem
• GE has 75 procurement systems
• CFO estimates that GE can save $100M per year if it
can empower each of the 75 procurement offices to
discover the terms and conditions negotiated by
his/her 74 counterparts at contract renewal time
• And demand most favored nation status
• Requires integrating 75 supplier databases!
• Enterprises also want to integrate parts, customers, lab
data,…
27

28. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Data Integration Challenge
• For each local data source, have to
— Ingest the source
— Perform transformations (e.g. $ to Euros)
— Perform data cleaning (a rule of thumb is at
least 10% of your data is wrong or missing)
— Perform schema integration
— Perform deduplication (GE’s problem)
— Find “golden values” in clusters of duplicates
28

29. Big Data, Disruption and the 800 Pound Gorilla in the Corner
At Scale
• GE has about 10M supplier records
• Toyota Motor Europe has about 30M customer
records
• Do not even think about naïve algorithms in python!!!!
29

30. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Traditional Solution
• Extract, Transform and Load (ETL) packages plus
Master Data Management (MDM) tools
• Brought to you by Informatica, IBM, Talend, Knime, …
• ETL requires too much manual programmer effort
• And is architected wrong
• MDM does not scale
• Rule system issues
30

31. Big Data, Disruption and the 800 Pound Gorilla in the Corner
MDM
● Does “match/merge” with rules
• A human can grok about 500 rules….
● A GE classification problem
• 20M transaction records
• To be classified into a pre-existing hierarchy
● GE wrote 500 rules
• Which classified 2M transactions
• What about the other 18M?
31

32. Big Data, Disruption and the 800 Pound Gorilla in the Corner
A Better Solution — Tamr
● Uses ML for schema integration, deduplication
and golden value resolution
• E.g. Used result of 500 rules as training data for an
ML model which classified the rest of the GE
transactions
• Dedup is not N ** 2 (that would be death)
32

33. Big Data, Disruption and the 800 Pound Gorilla in the Corner
The Future
• Lots of startups in this space
• Some oriented toward “data preparation”
• A few focused on enterprise data integration
• Some focused on text
• Some focused on deep learning
• The wild, wild west
• Hold onto your seat belt
33

34. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Summary
• ML will be omnipresent
• Some deep learning
• Some conventional ML
• Complex analytics (data science) will replace
business intelligence
• As soon as we can train enough data scientists
• Both will get nowhere without good data
• Requires data integration at scale
• The 800 pound gorilla
34

35. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Questions?
35

36. Big Data, Disruption and the 800 Pound Gorilla in the Corner
Thank you!