Big Data and NoSQL continue to make headlines everywhere. However, most of what has been written about these topics is focused on the hardware, services, and scale out. But what about a Big Data and NoSQL Strategy, one that supports your business strategy? Virtually every major organization thinking about these data platforms is faced with the challenge of figuring out the appropriate approach and the requirements. This presentation will provide guidance on how to think about and establish realistic Big Data management plans and expectations. We will introduce a framework for evaluating the various choices when it comes to implementing and succeeding with Big Data/NoSQL and show how to demonstrate a sample use case.
1. • Big Data
could
know us
better than
we know
ourselves
– Dan
Gardner
• We'll see this as the
time in history wh
the world's
information was
transformed from
inert, passive stat
and put into a
unified system th
brings that
information alive
– Michael Nielsen
ow have a
ce to en
me the
of our
nowledge
rse, one an
onstantly e,
figures
to match
eeds
hael S.
one
at
A Framework for Implementing
NoSQL, Hadoop
• N • Today a street stall in Mumbai can access more
b information, maps, statistics, academic papers, price
n trends, futures markets, and data than a U.S.
c President could only a few decades ago
– – Juan Enriquez
ot everything that can
e counted counts, and
ot everything that
ounts can be counted
Albert Einstein
Big Data and NoSQL continue to make headlines everywhere.
However, most of what has been written about these topics is
focused on the hardware, services, and scale out. But what about
a Big Data and NoSQL Strategy, one that supports your business
strategy? Virtually every major organization thinking about these
data platforms is faced with the challenge of figuring out the
appropriate approach and the requirements. This presentation will
provide guidance on how to think about and establish realistic Big
Data management plans and expectations. We will introduce a
framework for evaluating the various choices when it comes to
implementing and succeeding with Big Data/NoSQL and show
how to demonstrate a sample use case.
Takeaways:
• A Framework for evaluating Big Data techniques
• Deciding on a Big Data platform – How do you know which one
is a good fit for you?
• The means by which big data techniques can complement
existing data management practices
• The prototyping nature of practicing big data techniques
• The distinct ways in which utilizing Big Data can generate
business value
Date:
Time:
Presenter:
June 9, 2015
2:00 PM ET/11:00AM PT
PeterAiken, Ph.D. & Josh Bartels
• Soon we will salt the oceans, the land, and the sk
with uncounted numbers of sensors invisible to th
eyes but visible to one another
• We n – Esther Dyson
chan
beco
center
own k
unive
that c
recon
itself
our n
– Mic
Mal
• We've reached a tipping point in history: today more y
data is being manufactured by machines, servers, e
and cell phones, than by people
– Michael E. Driscoll
• Every century, a new technology-steam power,
electricity, atomic energy, or microprocessors-has
swept away the old world with a vision of a new one.
Today, we seem to be entering the era of Big Data
– Michael Coren
1Copyright 2015 by Data Blueprint Slide #
3. Steven MacLauchlan
• 10 years of experience in Application
Development and Data Modeling with a
focus on Healthcare solutions.
• Delivers tailored data management
solutions that provide focus on data’s
business value while enhancing clients’
overall capability to manage data
• Certified Data Management Professional (CDMP)
• Computer Science degree from Virginia Commonwealth
University
• Most recent focus: Understanding emerging
data modeling trends and how these can
best be leveraged for the Enterprise.
3Copyright 2015 by Data Blueprint Slide #
4. Get Social With Us!
Live Twitter Feed
Join the conversation!
Follow us:
@datablueprint
@paiken
Ask questions and submit
your comments: #dataed
Like Us on Facebook
www.facebook.com/
datablueprint
Post questions and comments
Find industry news, insightful
content
and event updates.
Join the Group
Data Management &
Business Intelligence
Ask questions, gain insights
and collaborate with fellow
data management
professionals
4Copyright 2015 by Data Blueprint Slide #
5. Peter Aiken, Ph.D.
• 30+ years in data management
• Repeated international recognition
• Founder, Data Blueprint (datablueprint.com)
• Associate Professor of IS (vcu.edu)
• DAMA International (dama.org)
• 9 books and dozens of articles
• Experienced w/ 500+ data
management practices
• Multi-year immersions:
– US DoD
– Nokia
– Deutsche Bank
– Wells Fargo
– Walmart
– …
• DAMA International President 2009-2013
• DAMA International Achievement Award 2001 (with
Dr. E. F. "Ted" Codd
• DAMA International Community Award 2005
PETERAIKEN WITH JUANITABILLINGS
F OR EW O RD B Y J O H N B OTTEGA
MONETIZING
DATA M AN AGEM EN T
Unlocking the Value in Your Organization’s
Most Important Asset.
TheCaseforthe
Chief ta fficer
Recasting uite erage
Your Most aluable A
Peter Aikenand
Michael Gorman
5Copyright 2015 by Data Blueprint Slide #
6. Josh Bartels
• Data management consultant and
leader
– Over (10) years of experience
– Multiple industries (Finance, Defense,
Insurance)
• Certifications
– Certified Data Management
Professional (CDMP)
– Project Manager (PMP)
– Data Vault 2.0 Practitioner (CDVP2)
• Education
– Masters in Business Administration
– Masters in Information Systems
• Current Efforts
– focus on the creation and migration to
new data platforms for clients in the
financial and insurance industries.
6Copyright 2015 by Data Blueprint Slide #
7. Presented by Peter Aiken, Ph.D., Josh Bartels, Steven MacLauchlan
A Framework for Implementing
NoSQL, Hadoop
Demystifying Big Data 2.0: Developing the Right
Approach for Implementing Big Data Techniques
7Copyright 2015 by Data Blueprint Slide #
8. A Framework for Implementing NoSQL, Hadoop
Demystifying Big Data 2.0: Developing the Right Approach for Implementing Big Data Techniques
• Big Data Context
– We are using the wrong vocabulary to discuss this topic
• More Precise Definitions
– Framework
– Non Von Neuman Architectures
– Hadoop/Nosql
• Big Data
– Historical Perspective
• Big Data Approach
– Crawl, Walk, Run
• Framework Examples
– Social
– Operational BWB
• Take Aways and Q&A
Tweeting now at: #dataed
8Copyright 2015 by Data Blueprint Slide #
9. A Framework for Implementing NoSQL, Hadoop
Demystifying Big Data 2.0: Developing the Right Approach for Implementing Big Data Techniques
• Big Data Context
– We are using the wrong vocabulary to discuss this topic
• More Precise Definitions
– Framework
– Non Von Neuman Architectures
– Hadoop/Nosql
• Big Data
– Historical Perspective
• Big Data Approach
– Crawl, Walk, Run
• Framework Examples
– Social
– Operational BWB
• Take Aways and Q&A
Tweeting now at: #dataed
10Copyright 2015 by Data Blueprint Slide #
10. Myth #1: Big Data has a clear definition
Fact:
• The term is used so often
and in many contexts that
its meaning has become
vague and ambiguous
• Industry experts and
scientists often disagree
http://articles.washingtonpost.com/2013-08-16/opinions/41416362_1_big-data-data-crunching-marketing-analytics
10Copyright 2015 by Data Blueprint Slide #
11. Big Data(has something to do with Vs - doesn't it?)
• Volume
– Amount of data
• Velocity
– Speed of data in and out
• Variety
– Range of data types and sources
• 2001 Doug Laney
• Variability
– Many options or variable interpretations confound analysis
• 2011 ISRC
•Vitality
–A dynamically changing Big Data environment in which analysis and predictive models
must continually be updated as changes occur to seize opportunities as they arrive
• 2011 CIA
•Virtual
– Scoping the discussion to only include online assets
• 2012 Courtney Lambert
• Value/Veracity
• Stuart Madnick (John Norris Maguire Professor of Information Technology, MIT Sloan School of
Management & Professor of Engineering Systems, MIT School of Engineering)
11Copyright 2015 by Data Blueprint Slide #
12. Defining Big Data
• Big Data are high-volume, high-velocity, and/or high-variety
information assets that require new forms of processing to
enable enhanced decision making, insight discovery
and process optimization.
– Gartner 2012
• Big data refers to datasets whose size is beyond the ability of
typical database software tools to capture, store, manage, and analyze.
– IBM 2012
• An all-encompassing term for any collection of data sets so large and complex that it
becomes difficult to process using on-hand data management tools or traditional data
processing applications
– Wikipedia 2014
• Shorthand for advancing trends in technology that open the door to a new approach
to understanding the world and making decisions.
– NY Times 2012
• The broad range of new and massive data types that have appeared over the last
decade
– Tom Davenport 2014
• Data of a very large size, typically to the extent that its manipulation and management
present significant logistical challenges.”
– Oxford English Dictionary 2014
• Big data is about putting the "I" back into IT.
– PeterAiken 2007
12Copyright 2015 by Data Blueprint Slide #
13. Big Data Techniques
• New techniques available to impact the productivity (order of
magnitude) of any analytical insight cycle that compliment,
enhance, or replace conventional (existing) analysis methods
• Big data techniques are currently characterized by:
– Continuous, instantaneously
available data sources
– Non-von Neumann
Processing (defined later in the presentation)
– Capabilities approaching
or past human comprehension
– Architecturally enhanceable
identity/security capabilities
– Other tradeoff-focused data processing
• So a good question becomes "where in our existing architecture
can we most effectively apply Big Data Techniques?"
13Copyright 2015 by Data Blueprint Slide #
14. Big Data Technologies by themselves, are a One Legged Stool
Governance is the major means
of preventing over reliance on
one legged stools!
14Copyright 2015 by Data Blueprint Slide #
15. The Big Data Landscape
Copyright Dave Feinleib, bigdatalandscape.com
15Copyright 2015 by Data Blueprint Slide #
18. Myth #2: Everyone should invest in Big Data
Fact:
• Not every company will
benefit from Big Data
• It depends on your size
and your ability
– Local pizza shop vs.
state-wide or national
chain
18Copyright 2015 by Data Blueprint Slide #
19. Big Data can create significant financial value across sectors
• Some (not all)
companies can
take advantage
of Big Data to
create value if
they want to
compete
20Copyright 2015 by Data Blueprint Slide #
20. A Framework for Implementing NoSQL, Hadoop
Demystifying Big Data 2.0: Developing the Right Approach for Implementing Big Data Techniques
• Big Data Context
– We are using the wrong vocabulary to discuss this topic
• More Precise Definitions
– Framework
– Non Von Neuman Architectures
– Hadoop/Nosql
• Big Data
– Historical Perspective
• Big Data Approach
– Crawl, Walk, Run
• Framework Examples
– Social
– Operational BWB
• Take Aways and Q&A
Tweeting now at: #dataed
20Copyright 2015 by Data Blueprint Slide #
21. Big Data = Big Spending
• Enterprises are spending wildly on Big Data but don’t
know if it’s worth it yet (Business Insider, 2012)
• Big Data Technology Spending Trend:
• 83% increase over the next 3 years (worldwide):
– 2012: $28 billion
– 2013: $34 billion
– 2016: $232 billion
• Caution:
– Don’t fall victim to SOS (Shiny Object
Syndrome)
– A lot of money is being invested but
is it generating the expected return?
– Gartner Hype Cycle suggests results
are going to be disappointing http://www.businessinsider.com/enterprise-big-data-spending-2012-11#ixzz2cdT8shhe
http://www.inc.com/kathleen-kim/big-data-spending-to-increase-for-it-industry.html
http://www.gartner.com/DisplayDocument?id=2195915&ref=clientFriendlyUrl
21Copyright 2015 by Data Blueprint Slide #
22. Who wrote this … ?
23
Copyright 2015 by Data Blueprint
• In considering any new
subject, there is
frequently a tendency
first to overrate what
we find to be already
interesting or
remarkable, and
secondly - by a sort of
natural reaction - to
undervalue the true
state of the case.
• AugustaAda King,
Countess of Lovelace - aka
Ada Lovelace, publisher of
the first computing program
23. Gartner Five-phase Hype Cycle
http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp
Peak of Inflated Expectations: Early publicity produces a number of
success stories—often accompanied by scores of failures. Some
companies take action; many do not.
Trough of Disillusionment: Interest wanes as experiments and implementations fail to deliver. Producers of the
technology shake out or fail. Investments continue only if the surviving providers improve their products to the
satisfaction of early adopters.
Technology Trigger: A potential technology breakthrough kicks things off. Early proof-of-concept stories and media interest
trigger significant publicity. Often no usable products exist and commercial viability is unproven.
Slope of Enlightenment: More instances of how the technology can benefit the
enterprise start to crystallize and become more widely understood. Second- and third-
generation products appear from technology providers. More enterprises fund pilots;
conservative companies remain cautious.
Plateau of Productivity: Mainstream adoption starts to
take off. Criteria for assessing provider viability are more
clearly defined. The technology’s broad market
applicability and relevance are clearly paying off.
23Copyright 2015 by Data Blueprint Slide #
24. Gartner Hype Cycle
"A focus on big data is not a substitute for the
fundamentals of information management."
24Copyright 2015 by Data Blueprint Slide #
25. 2012 Big Data in Gartner’s Hype Cycle
25Copyright 2015 by Data Blueprint Slide #
26. 2013 Big Data in Gartner’s Hype Cycle
26Copyright 2015 by Data Blueprint Slide #
27. 2014 Big Data in Gartner’s Hype Cycle
27Copyright 2015 by Data Blueprint Slide #
28. Big Data Gartner Hype Cycle
Copyright 2015 by Data Blueprint Slide #
29
29. Myth #3: Big Data is innovative
Fact:
• Big Data techniques are
innovative
• ROI and insights depend
on the size of the business
and the amount of data
used and produced, e.g.
– Local pizza place vs. Papa
John’s
– Retail
29Copyright 2015 by Data Blueprint Slide #
30. My Barn must pass a foundation inspection
• Before further construction can proceed
• No IT equivalent in most organizations
30Copyright 2015 by Data Blueprint Slide #
31. Frameworks
• A system of ideas
for guiding
analyses
• A means of
organizing project
data
• Data integration
priorities decision
making
framework
• A means of
assessing
progress
8 31Copyright 2015 by Data Blueprint Slide #
32. "There’s now a blurring between the storage world and the memory world"
• Faster processors outstripped
not only the hard disk, but main
memory
– Hard disk too slow
– Memory too small
• Flash drives remove both
bottlenecks
– Combined Apple and Yahoo have
spend more than $500 million to
date
• Make it look like traditional
storage or more system
memory
– Minimum 10x improvements
– Dragonstone server is 3.2 tb flash
memory (Facebook)
• Bottom line - new capabilities!
8 32Copyright 2015 by Data Blueprint Slide #
33. Non-von Neumann Processing/Efficiencies
• von Neumann
bottleneck
(computer science)
– "An inefficiency inherent in
the design of any von
Neumann machine that
arises from the fact that
most computer time is
spent in moving
information between
storage and the central
processing unit rather than
operating on it"
[http://encyclopedia2.thefreedictionary.com/von+Neumann+bottleneck]
• Michael Stonebraker
– Ingres (Berkeley/MIT)
– Modern database
processing is
approximately 4%
efficient
• Many big data
architectures are
attempts to address
this, but:
– Zero sum game
– Trade characteristics
against each other
• Reliability
• Predictability
– Google/MapReduce/
Bigtable
– Amazon/Dynamo
– Netflix/Chaos Monkey
– Hadoop
– McDipper
• Big data techniques
exploit non-von
Neumann processing
8 33Copyright 2015 by Data Blueprint Slide #
35. One of Data Blueprint's Big Data Clusters
8 35Copyright 2015 by Data Blueprint Slide #
36. <-Feedback
Exploitable
Insight
• Patterns/objects,
hypotheses emerge
– What can be observed?
• Operationalizing
– The dots can be
repeatedly connected
Analytics Insight Cycle
Exis&ng
Knowledge
/base
• Things are happening
– Sensemaking
techniques address
"what" is happening?
• Patterns/objects,
hypotheses emerge
– What can be observed?
• Operationalizing
– The dots can be
repeatedly connected
– "Big Data" contributions
are shown in orange
• Margaret Boden's
computational
creativity
– Exploratory
– Combinational
– Transformational
Volume
Variety
Velocity
Potential/
actual
insights
Pattern/Object
Emergence
Analytical
bottleneck
8 36Copyright 2015 by Data Blueprint Slide #
37. Big Data: Two prominent use cases
• Sandwich offers a good analogy
of the big data and existing
technologies
• Landing Zone (less expensive)
– Especially useful in cases were data
is highly disposable
• Existing technologies are the
– Contents sandwiched and
complemented landing zone and
archival capabilities
• Archiving/Offloading (less need
for structure)
– "Cold" transactional and analytic
data
Adapted from Nancy Kopp:
http://ibmdatamag.com/2013/08/relishing-the-big-data-burger/
Landing Zone
Archiving Offloading
Existing
Data Architectural
Processing
8 37Copyright 2015 by Data Blueprint Slide #
38. What is NoSQL?
• Commonly interpreted as "Not Only SQL
• Broad class of database management technologies that
provide a mechanism for storage and retrieval of data that
doesn’t follow traditional relational database methodology.
• Motivations
– Simplicity of design
– Horizontal scaling
– Finer control over availability of the data.
• The data structures used by NoSQL databases differ from
those used in relational databases, making some
operations faster in NoSQL
and others faster in relational
databases.
8 38Copyright 2015 by Data Blueprint Slide #
39. What is Hadoop?
• A data storage and processing
system, that runs on clusters of commodity servers.
• Able to store any kind of data in its native format.
• Perform a wide variety of analyses and transformations.
• Store terabytes, and even petabytes, of data
inexpensively.
• Handles hardware and system failures automatically,
without losing data or interrupting data analyses.
• Critical components of Hadoop:
– HDFS- The Hadoop Distributed File System is the storage system
for a Hadoop cluster, responsible for distribution of data across the
servers.
– Mapreduce- The inner workings of Hadoop that allows for distributed
and parallel analytical job execution.
40Copyright 2015 by Data Blueprint Slide #
40. Why NoSQL? Why Hadoop?
• Large number of users (read: the internet)
• Rapid app development and deployment
• Large number of mission critical writes (sensors/etc)
• Small, continuous reads and writes, especially where
“Consistency” is less important (social networks)
• Hadoop solves the hard scaling problems caused by large
amounts of complex data.
• As the amount of data in a cluster grows,
new servers can be added to a Hadoop
cluster incrementally and inexpensively
to store and analyze it.
40Copyright 2015 by Data Blueprint Slide #
41. Hadoop Use Cases in the Real World
• Risk Modeling
• Customer Churn Analysis
• Recommendation Engine
• Ad Targeting
• Point of Sale Transaction Analysis
• Social Sentiment on Social Media
• Analyzing network data to predict failure
• Threat analysis
• Trade Surveillance
41Copyright 2015 by Data Blueprint Slide #
43. 44
Copyright 2015 by Data Blueprint
• Data analysis struggles with the social
– Your brain is excellent at social cognition - people can
• Mirror each other’s emotional states
• Detect uncooperative behavior
• Assign value to things through emotion
– Data analysis measures the quantity of social
interactions but not the quality
• Map interactions with co-workers you see during work days
• Can't capture devotion to childhood friends seen annually
– When making (personal) decisions about social
relationships, it’s foolish to swap the amazing machine
in your skull for the crude machine on your desk
• Data struggles with context
– Decisions are embedded in sequences and contexts
– Brains think in stories - weaving together multiple
causes and multiple contexts
– Data analysis is pretty bad at
• Narratives / Emergent thinking / Explaining
• Data creates bigger haystacks
– More data leads to more statistically significant
correlations
– Most are spurious and deceive us
– Falsity grows exponentially greater amounts of data
we collect
• Big data has trouble with big problems
– For example: the economic stimulus debate
– No one has been persuaded by data to switch sides
• Data favors memes over masterpieces
– Detect when large numbers of people take an instant
liking to some cultural product
– Products are hated initially because they are unfamiliar
• Data obscures values
– Data is never raw; it’s always structured according to
somebody’s predispositions and values
Some Big Data Limitations
44. Myth #4: Big Data is just another IT project
Copyright 2013 by Data Blueprint
Fact:
• Big Data is not your typical IT
project
– Does not answer typical IT questions
– Trend analysis, agile, actionable, etc.
– Fundamentally different approach
• Big Data Projects are exploratory
• Big Data enables new capabilities
• Big Data can be a disruptive
technology
• It might sound simple but that
doesn’t mean it’s easy
• Beware of SOS (Shiny Object
Syndrome)
44
48. ("Whereas of the Plague")
Plague Peak
When is it happening?
Copyright 2015 by Data Blueprint
48
49. Black Rats or Rattus Rattus
Why is it happening?
50
Copyright 2015 by Data Blueprint
50. What Will Happen? What will happen?
51
Copyright 2015 by Data Blueprint
51. Formalizing Data Management
• Defend the Realm:
The authorized history of MI5
by Christopher Andrew
• World War I
• 1914
• At war with much
of Europe
• 14,000,000 Germans living
in the United Kingdom
• How to efficiently and
effectively manage
information on that many
individuals?
• The Security Service is responsible for "protecting
the UK against threats to national security from
espionage, terrorism and sabotage, from the activities
of agents of foreign powers, and from actions intended
to overthrow or undermine parliamentary democracy by
political, industrial or violent means."
51Copyright 2015 by Data Blueprint Slide #
52. “As a final thought, how about a machine that
would send, via closed-circuit television, visual and
oral information needed immediately at high-level
conferences or briefings? Let’s say that a group of
senior officers are contemplating a covert action
program for Afghanistan. Things go well until
someone asks “Well, just how many schools are
there in the country, and what is the literacy rate?”
No one in the room knows. (Remember, this is an
imaginary situation). So the junior member present
dials a code number into a device at one end of the
table. Thirty seconds later, on the screen overhead,
a teletype printer begins to hammer out the
required data. Before the meeting is over, the group
has been given, through the same method, the
names of countries that have airlines into
Afghanistan, a biographical profile of the Soviet
ambassador there, and the Pakistani order of battle
along the Afghanistan frontier. Neat, no?”
• Predicted use of
not just
computing in the
intelligence
community
• Also forecast
predictive
analytics
• Accompanying
privacy
challenges
52Copyright 2015 by Data Blueprint Slide #
53. A Framework for Implementing NoSQL, Hadoop
Demystifying Big Data 2.0: Developing the Right Approach for Implementing Big Data Techniques
• Big Data Context
– We are using the wrong vocabulary to discuss this topic
• More Precise Definitions
– Framework
– Non Von Neuman Architectures
– Hadoop/Nosql
• Big Data
– Historical Perspective
• Big Data Approach
– Crawl, Walk, Run
• Framework Examples
– Social
– Operational BWB
• Take Aways and Q&A
Tweeting now at: #dataed
53Copyright 2015 by Data Blueprint Slide #
54. http://articles.washingtonpost.com/2013-08-16/opinions/41416362_1_big-data-data-crunching-marketing-analytics
Copyright 2013 by Data Blueprint
Myth #6: Big Data provides all the Answers
Fact:
• Big Data does not mean the end of
scientific theory
• Be careful or you’ll end up with
spurious correlations
– Don’t just go fishing for correlations and
hope they will explain the world
• To get to the WHY of things, you
need ideas, hypotheses and theories
• Having more data does not
substitute for thinking hard,
recognizing anomalies and exploring
deep truths
• You need the right approach
54
56. • Identify business opportunity
Copyright 2013 by Data Blueprint
• How can data be leveraged in
exploring
– External market place
• Analyze opportunities and threats
– Internal efficiencies
• Analyze strengths and weaknesses
56
57. Example: 2012 Olympic Summer Games
Copyright 2013 by Data Blueprint
1. Volume: 845 million FB users averaging 15 TB
+ of data/day
2. Velocity: 60 GB of data per second
3. Variety: 8.5 billion devices connected
4. Variability: Sponsor data, athlete data, etc.
5. Vitality: Data Art project “Emoto”
6. Virtual: Social media
57
58. • Based on my 6 V analysis, do I need a Big Data solution
Copyright 2013 by Data Blueprint
or does my current BI solution address my business
opportunity?
– Do the 6 Vs indicate general Big Data characteristics?
– What are the limitations of my current Bi environment?
(Technology constraint)
– What are my budgetary restrictions? (Financial constraint)
– What is my current Big Data knowledge base? (Knowledge
constraint)
58
59. • MUST have both
Foundational and
Technical practice
expertise
60
Copyright 2013 by Data Blueprint
61. • Data Strategy
Copyright 2013 by Data Blueprint
• Data Governance
• Data Architecture
• Data Education
61
62. • Data Quality
Copyright 2013 by Data Blueprint
• Data Integration
• Data Platforms
• BI/Analytics
62
63. • Needs to be actionable
• Generally well understood by
business
• Document what has been learned
Copyright 2013 by Data Blueprint
63
64. • Perfect results are not
necessary
• Reiterate and refine
• Iterative process to
reach decision point
• Use as feedback for
next exploration
Copyright 2013 by Data Blueprint
64
66. Myth #7: You need Big Data for Insights
Fact:
• Distinction between Big Data and
doing analytics
– Big Data is defined by the technology stack
that you use
– Big Data is used for predictive and
prescriptive analytics
• Use existing data for reporting, figure
out bottlenecks and optimize current
business model
• Understand how is your data
structured, architected and stored
Copyright 2013 by Data Blueprint
66
67. A Framework for Implementing NoSQL, Hadoop
Demystifying Big Data 2.0: Developing the Right Approach for Implementing Big Data Techniques
• Big Data Context
– We are using the wrong vocabulary to discuss this topic
• More Precise Definitions
– Framework
– Non Von Neuman Architectures
– Hadoop/Nosql
• Big Data
– Historical Perspective
• Big Data Approach
– Crawl, Walk, Run
• Framework Examples
– Social
– Operational BWB
• Take Aways and Q&A
68Copyright 2015 by Data Blueprint Slide #
Tweeting now at: #dataed
68. Social Sentiment Analysis
• One of the burgeoning areas
for use of Big Data / Hadoop
platforms.
• Allows for the landing of
multiple sources of
unstructured data. (Twitter,
Facebook, Linked In, etc.)
• Data than can be analyzed
with algorithms looking for
keywords that determine
positive/negative feedback
Copyright 2013 by Data Blueprint
69
69. Operational Use
• Utilize real time pricing data from multiple sources to dynamically
update the pricing for books in the Amazon Marketplace.
• Ingested data from multiple sources looking for real time changes
in price.
• Would apply predictive model to determine best price point and set
price of their books on the marketplace.
• Increased conversion rate, but created a race to the bottom
situation if not monitored
Copyright 2013 by Data Blueprint
79
70. Healthcare Example: Patient Data
Copyright 2013 by Data Blueprint
• Clinical data:
– Diagnosis/prognosis/treatment
– Genetic data
• Patient demographic data
• Insurance data:
– Insurance provider
– Claims data
• Prescriptions & pharmacy information
• Physical fitness data
– Activity tracking through
smartphone apps & social media
• Health history
• Medical research data
70
71. http://www.forbes.com/sites/xerox/2013/09/27/big-data-boosts-customer-loyalty-no-really/
Copyright 2013 by Data Blueprint
Retail Example: Loyalty Programs & Big Data
• Companies need to understand current wants and needs AND
predict future tendencies
• Customer -> Repeat Customer -> Brand Advocate
• Customer loyalty programs & retention strategies
– Track what is being purchased and how often
– Coupons based on purchasing history
– Targeted communications, campaigns & special offers
– Social media for additional interactions
– Personalize consumer interactions
• Customer purchase history influences
product placements
– Retailers rapidly respond to consumer demands
– Product placements, planogram optimization, etc.
71
72. References
Copyright 2013 by Data Blueprint
• The Human Face of Big Data, Rick Smolan & Jennifer Erwitt, First Edition edition (November
20, 2012)
• McKinsey: Big Data: The next frontier for innovation, competition and productivity
(http://www.mckinsey.com/insights/business_technology/
big_data_the_next_frontier_for_innovation?p=1)
• The Washington Post: Five Myths about Big Data (http://articles.washingtonpost.com/
2013-08-16/opinions/41416362_1_big-data-data-crunching-marketing-analytics)
• Gartner: Gartner’s 2013 Hype Cycle for Emerging Technologies Maps Out Evolving
Relationship Between Humans and Machines (http://www.gartner.com/newsroom/id/
2575515)
• The New York Times | Opinion Pages: What Data Can’t Do (http://www.nytimes.com/
2013/02/19/opinion/brooks-what-data-cant-do.html?_r=1&)
• CIO.com: Five Steps for How to Better Manage Your Data (http://www.cio.com.au/article/
429681/five_steps_how_better_manage_your_data/)
• Business Insider: Enterprises Aren’t Spending Wildly on ‘Big Data’But Don’t Know If It’s
Worth It Yet (http://www.businessinsider.com/enterprise-big-data-
spending-2012-11#ixzz2cdT8shhe)
• Inc.com: Big Data, Big Money: IT Industry to Increase Spending (http://www.inc.com/
kathleen-kim/big-data-spending-to-increase-for-it-industry.html)
• Forbes: Big Data Boosts Customer Loyalty. No, Really. (http://www.forbes.com/sites/xerox/
2013/09/27/big-data-boosts-customer-loyalty-no-really/)
72
73. Data Management Maturity
July 14, 2015 @ 2:00 PM ET/11:00 AM PT
Trends in Data Modeling
August 11, 2015 @ 2:00 PM ET/11:00 AM PT
Sign up here:
www.datablueprint.com/webinar-schedule
or www.dataversity.net
Upcoming Events
Copyright 2013 by Data Blueprint
73
74. 10124 W. Broad Street, Suite C
GlenAllen, Virginia 23060
804.521.4056
75. Copyright 2013 by Data Blueprint
77
Potential Tradeoffs:
CAP theorem: consistency, availability and partition-tolerance
Small datasets can be both consistent & available
Partition
(Fault)
Tolerance
AvailabilityConsistency
Atomicity
Consistency
Isolation
Durability
Basic
Availability
Soft-state
Eventual consistency
77. http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation?p=1
Copyright 2013 by Data Blueprint
5 Ways in which Data creates Business Value
1. Information is transparent
and usable at much higher
frequency
2. Expose variability and
boost performance
3. Narrow segmentation of
customers and more
precisely tailored products
or services
4. Sophisticated analytics and
improved decision-making
5. Improved development of
the next generation of
products and services
77
78. • We are at an inflection point: The
sheer volume of data generated,
stored, and mined for insights has
become economically relevant to
businesses, government, and
consumers (McKinsey)
• We believe the same important
principles still apply:
– What problem are you trying to solve for
your business? Your solution needs to fit
your problem
– Doing data for (big) data’s sake is not going
to solve any problems
– Risk of spending a lot of money on chasing
Big Data that will realize little to no returns -
especially at this hype cycle stage
http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation?p=1
Why the Big Deal about Big Data?
80
Copyright 2013 by Data Blueprint
80. Take Aways-Big Data Context
Copyright 2013 by Data Blueprint
• Technology continues to evolve at
increasing speeds
• Big Data is here
– We have the potential to
create insights
• Spend wisely & strategically:
– Big Data is not going to solve
all your problems.
• Fact:
– Big Data is not for everyone
• Fact:
– Lack of a clear definition
• Hype Cycle:
– Current: Peak of Inflated Expectations
– Soon: Trough of Disillusionment
80
81. Take Aways: Big Data Challenges Today
Copyright 2013 by Data Blueprint
• Fact: Big Data techniques are innovative but
“Big Data” is not
• Challenges are both foundational and
technical, today as well as in 1600s
• Technology continues to advance rapidly (4
Vs)
• Challenges associated with Big Data are not
new:
– Well-known foundational data management issues
– Need to align data and business with rapidly
changing environment
– Duplicity, accessibility, availability
– Foundational business issues
81
82. Take Aways-Approach: Crawl, Walk, Run
Copyright 2013 by Data Blueprint
• Crawl:
– Identify business opportunity and
determine whether you truly need
a Big Data solution
• Walk:
– Apply a combination of
foundational and technical data
management practices.
Document your insights and
make sure they are actionable
• Run:
– Recycle and explore. Staying
agile allows you to be exploratory.
82
83. Take Aways-Design Principles: Foundational & Technical
Copyright 2013 by Data Blueprint
• Foundational data management
principles still apply
• Beware of SOS (Shiny Object
Syndrome)
• You must have a data strategy before
you can have a Big Data strategy
• Fact: You don’t need Big Data to gain
insights
• Big Data integration requirements evolve
from your strategy
• Fact: Bigger Data is not always better
83
84. Take Aways: In Summary
Copyright 2013 by Data Blueprint
• Big data techniques are innovative
but “Big Data” is not
• Big Data characteristics: 6 Vs
– Volume, Velocity, Variety, Variability, Vitality,
Virtual
• Approach: Crawl-Walk-Run
• Big Data challenges require solutions
that are based on foundational and
technical data management practices
• Beware of SOS (Shiny Object
Syndrome):
– Spend wisely and strategically
– Big Data is not going to solve all your
problems
84
85. Foundational Practice: Data Strategy
• Your data strategy must
align to your organizational
business strategy and
operating model
• As the market place
becomes more data-
driven, a data-focused
business strategy is an
imperative
• Must have data strategy
before you have a Big
Data strategy
Copyright 2013 by Data Blueprint
85
86. Data Strategy Considerations
• What are the questions that
you cannot answer today?
• Is there a direct reliance on
understanding customer
behavior to drive revenue?
• Do you have information
overload and are you trying to
find the signal in the noise?
• Which is more important:
– Establishing value from current
data assets/data reporting?
– Exploring Big Data
opportunities?
Copyright 2013 by Data Blueprint
86
87. Foundational Practice: Data Architecture
• Common vocabulary expressing
integrated requirements ensuring
that data assets are stored,
arranged, managed, and used in
systems in support of
organizational strategy [Aiken
2010]
• Most organizations have data
assets that are not supportive of
strategies
• Big question:
– How can organizations more
effectively use their information
architectures to support
strategy implementation?
90
Copyright 2013 by Data Blueprint
88. Data Architecture Considerations
• Does your current architecture for
BI and analytics support Big Data?
• Are you getting enough value out of
your current architecture?
• Can you easily integrate and share
information across your
organization?
• Do you struggle to extract the value
from your data because it is too
cumbersome to navigate and
access?
• Are you confident your data is
organized to meet the needs of
your business?
Copyright 2013 by Data Blueprint
88
89. Technical Practice: Data Integration
• A data-centric
organization requires
unified data
• Integrating data across
organizational silos
creates new insights
• It is also the biggest
challenge
• Big Data techniques can
be used to complement
existing integration efforts
Copyright 2013 by Data Blueprint
89
90. Data Integration Considerations
• The complexity of your data
integration challenge depends on
the questions you’re trying to
answer
• Integration requirements for Big
Data are dependent on the types of
questions you’re asking:
– Integration here may be more fuzzy than
discrete
– Integration is domain-based (based on
time, customer concept, geographic
distribution)
• Those requirements should evolve
from your strategy
Copyright 2013 by Data Blueprint
90
91. Technical Practice: Data Quality
• Quality is driven by fit for purpose
considerations
• Big Data quality is different:
– Basic
– Availability
– Soft-state
– Eventual consistency
• Directional accuracy is the goal
• Focus on your most important data
assets and ensure our solutions
address the root cause of any quality
issues – so that your data is correct
when it is first created
• Experience has shown that
organizations can never get in front of
their data quality issues if they only use
the ‘find-and-fix’ approach
Copyright 2013 by Data Blueprint
91
92. Data Quality Considerations
• Big Data is trying to be
predictive
• What are the questions you
are trying to answer?
– What level of accuracy are you
looking for?
– What confidence levels?
– Example: Do I need to know
exactly what the customer is
going to buy or do I just need to
know the range of products he/
she is going to choose from?
Copyright 2013 by Data Blueprint
92
93. Technical Practice: Data Platforms
• Do you want to measure
critical operational process
performance?
• No one data platform can
answer all your questions. This
is commonly misunderstood
and often leads to very
expensive, bloated and
ineffective data platforms.
• Understanding the questions
that need to be asked and how
to build the right data platform
or how to optimize an existing
one
Copyright 2013 by Data Blueprint
93
94. Data Platforms Considerations
• Commonalities between most big data
stacks with file storage, columnar store,
querying engine, etc.
• Big data stack generally looks the same
until you get into appliances
– Algorithms are built into appliance
themselves, e.g. Netezza, Teradata,
etc.)
• Ask these questions:
– Do you want insights on your
customer’s behavior?
– Do you need real-time customer
transactional information?
– Do you need historical data or just
access to the latest transactions?
– Where do you go to find the single
version of the truth about your
customers?
Copyright 2013 by Data Blueprint
94