1.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 1

2.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 2
Speakers
Dr. Michael Stonebraker
Co-Founder,
Tamr
Anthony Deighton
Chief Product Officer,
Tamr

3.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Blunder #1
Not Planning to Move Most EVERYTHING to the Cloud
3

4.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
It may take a decade, but it is the right thing to do
● Dewitt vignette
● Hamilton vignette
● Elasticity!!!
● Data will move easier than applications -- decision support first
4

5.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
YABUT...
5
Security
● Cloud security is
likely better than
yours
● Misconfiguration,
rogue employees
Cost
● Likely that
you are
cheating
Geographic
Restrictions
● Cloud guys
respect this
Legal
Restrictions
● Hopefully a
short term
problem
Other
Restrictions
● Your CEO
doesn’t
approve (see
item 11 to
come)

6.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
YABUT...
6
Where does App run?
● Decision support: move the app
● Other stuff:
○ Start with local deployment; move to remote data (SLOWLY!!!)
○ Migrate to cloud-native as you have resources, starting with the most
costly ones
○ This may be a lot of work and may take a decade or more
○ Issue is legacy code/hardware

7.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 7
Blunder #2
Not Planning for AI/ML to be Disruptive

8.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Blunder #2
Not Planning for AI/ML to be Disruptive
ML (whether deep or conventional) is getting much better
● Will displace workers with easy-to-explain jobs
● Think autonomous vehicles, automatic checkout, drone delivery, actuary
calculations
Likely to be disruptive
● You can be a disruptor or get disrupted - Your choice
● Think Uber/Lyft or taxis
8

9.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
So what to do?
9
Pay up to get some AI/ML experts
● They are in short supply and very expensive
● Don’t contract this out (See Blunder #8)
Get going on the coming arms race
● You will be a winner or a loser in a winner-take-all sweepstakes

10.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 10
Blunder #3
Not Solving your REAL Data Science Problem

11.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Blunder #3
Not Solving your REAL Data Science Problem
Typical data scientist spends 90+% of his/her time on data discovery, data
integration and data cleaning
● Irobot vignette
● Merck vignette
Nobody quotes less than 80%!!!
● Without clean data ML is worthless!!!
○ More accurately without “clean enough” data, ML is worthless
Obvious directive: Get a strategy in place to do this
● Start by giving Chief Data Officer (CDO) read access to ALL enterprise data!
11

12.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 12
Blunder #4
Belief that Traditional Data Integration
Techniques Will Solve Issue #3

13.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Blunder #4
Belief that Traditional Data Integration Techniques Will
Solve Issue #3
Exact Transformation and Load
(Available from a variety of vendors)
13
Master Data Management
(Also available from the usual suspects)

14.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
ETL
What’s attempted:
● Decide what data sources to
integrate (top dow)
● Build a global data model (up front)
● For each data source
○ Send a programmer to interview
the data set owner
○ He then builds an extractor, data
cleaning routines (in a proprietary
scripting language)
○ And loads data into the global
schema
14
Why it doesn’t work:
● I have never seen this technique work for
more than 20 data sources
○ Too human intensive
● Building a global schema upfront is way
too different at scale
○ Remember enterprise wide data models
from 15-20 years ago...
● Most enterprises I know have way more
than 20 data sources
○ Merck has 4000+/- Oracle data
bases
○ A data lake
○ Countless files
○ And data from the web is also
important

15.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
MDM
● Once you have run ETL, you need “match/merge”
● MDM suggests building “golden records” by
○ Implementing match rues (e.g. two entities are the same if they have the same
address)
○ Implementing merge rules (e.g. take the most recent value and ignore older ones)
Doesn’t Scale!
● GE classification problem: 20M spend transactions to be classified
into a pre-built hierarchy
● 500 rules classified only 10% of the spend transaction
15

16.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
So what to do?
At scale, you need a solution that leverages ML and statistics
● OK to use rules to generate training data
● That’s what Tamr did on the GE problem
16
+

17.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 17
Blunder #5
Belief that Data Warehouses will Solve all your Problems

18.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Blunder #5
Belief that Data Warehouses will Solve all your Problems
18
Data warehouses are good at customer facing structured data
FROM A FEW DATA SOURCES
● But not text, images, video, …
● Use the technology for what it is good for
○ Do not perform unnatural acts!
○ And get rid of the “high price spread”, if you bought into it
○ And remember that your warehouse will move to the cloud (see
Blunder #1)

19.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 19
Blunder #6
Belief that Hadoop/Spark will Solve all your Problems

20.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Blunder #6
Belief that Hadoop/Spark will Solve all your Problems
20
● Hadoop/Spark is not very good at anything
○ E.g. Spark/SQL is not competitive (but getting better)
○ E.g. Spark/Streaming is not competitive (last time I looked)
● Use “best of breed” not “lowest common denominator” -- at least for your
“secret sauce”
○ This is a universal blunder -- desire to use only one vendor
○ Hadoop/Spark is not very good at anything
● And…
○ Spark/Hadoop is useless on Blunders #3 and #4 (i.e. data integration)

21.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
So what to do with your Hadoop/Spark cluster?
● Repurpose it or a Data Lake
● Repurpose it for Data Integration
● Throw it Away
○ Hardware lifetime is 3 years (maybe)
○ Remember Blunder #1
21

22.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 22
Blunder #7
Belief that Data Lakes will Solve all your Problems

23.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Blunder #7
Belief that Data Lakes will Solve all your Problems
23
Conventional Wisdom
Just load all your data into a “data
lake” and you will be able to
correlate all data sets
Important Fact (Tattoo this on
your Brain):
Independently constructed data
sets are never “plug compatible”

24.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Why?
● Schemas don’t match
○ You call it salary; I call it
wages
● Units don’t match
○ You use Euros; I use $$$
● Semantics don’t match
○ My salaries are gross before
taxes; yours are net after
taxes with a lunch allowance
24
● Time granularity doesn't match
○ You have annual data; I have
monthly data
● Data is dirty
○ 99 means null (sometimes)
○ Null means “data missing” or
“data not allowed” or...
● Duplicates must be removed
○ And there are no keys
○ I am Mike Stonebraker in
one data set; M.R.
Stonebreaker in a second
one

25.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
The Net Result
● Your analytics will be garbage
○ “GIGO”
● Your ML models will fail
○ I.e. produce garbage
○ Again “GIGO”
25

26.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
So what to do?
● You don’t have a data lake; you have a data swamp
● Need a data curation system
○ Which will solve the aforementioned problems
○ And this will not be trivial!!
● Traditional technology likely to fail (See Blunder #4)
● This is an 800 pound gorilla
○ Make sure you put your best people on it!!!!
○ Chances are your in-house solution is crap
○ Use modern technology (from startups) not your “home brew”
● If you want the best technology, you have to deal with startups!!!!
26

27.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 27
Blunder #8
Outsourcing your new stuff to Palantir, IBM, Mu Sigma

28.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Blunder #8
Outsourcing your new stuff to Palantir, IBM, Mu Sigma
28
● Typical enterprise spends 95% of its IT resources keeping current
(legacy) code running
○ i.e. Maintenance
○ Most are dug in pretty deep
○ Often have the best people “keeping the lights on”
● “Shiny new stuff” gets outsourced
○ Often because here is no appropriate talent internally

29.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
This is a catch 22
● Your maintenance is boring!
○ So creative people quit
○ So there is no good talent to work on the new stuff
○ And you can’t hire great talent (Takes great people to hire great people)
● Your new stuff is your “secret sauce” over the next decade or so…
○ Please don’t outsource it. This is long-term suicide
○ Instead outsource the diddly-crap (e-mail et. al.)
○ Software is your secret sauce -- invest in your own people
29

30.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
So what to do?
1. Start by solving Blunder #2
(Not planning for AI/ML to change most everything)
1. Outsource the borning maintenance
2. Cancel the Palantir contract
30

31.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 31
Blunder #9
Succumbing to the “Innovator’s Dilemma”

32.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Blunder #9
Succumbing to the “Innovator’s Dilemma”
32
● Must read book by Clayton
Christensen
● Stream shovel example
○ Cable stream shovels - big payload
○ Hydraulics - much safer, but low
payload
● Used for “small jobs”
○ Payloads increased and hydraulics
won
○ Cable guys went out of business

33.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Net-Net
● Have to be willing to give up your current business model
● And reinvent yourself
● Possibly losing some current customers in the process
○ Otherwise, you go out of business in the long run
○ Taxi licenses in Cambridge have gone from $700k to $10k
33

34.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 34
Blunder #10
Not Paying Up for a Few “Rocket Scientists”

35.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Blunder #10
Not Paying Up for a Few “Rocket Scientists”
35
● They will be your guiding light to avoiding these blunders
● They will be “off scale”
○ Your HR folks won’t like what you have to pay
● Chances are they will be weird
○ E.g. no shoes, no socks, no tie, feet on the table, ...
● Please don’t drive them away!
○ As Citibank did to one of my Berkeley students a while ago

36.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 36
Blunder #11 (Bonus)
Working for a Company That is not Trying to
do Something about the “Sins of the Past”

37.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Blunder #11 (Bonus)
Working for a Company That is not Trying to do
Something about the “Sins of the Past”
37
If you work for a company that is succumbing to (even one) of these blunders
then:
1. You should be fixing it
a. Be part of the solution, not part of the problem
2. Or looking for a new employer
a. Tamr is hiring!

38.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021
Questions?
38

39.
How to Avoid the 10 Big Data Blunders - Best Practices for Success in 2021 39
Thank You!
To learn more about Tamr visit tamr.com
You’ll receive the 10 Big Data Analytics
Blunders Infographic via email.