With the computer revolution vast amount of digital data has become available. With the Internet and smart connected product, the data is growing exponentially. It is estimated that every year, more data is generated than all history prior. And this has repeated over several years. With all this data, it becomes a platform for something new of its own. In this lecture, we look at what big data is and look at several examples of how to use data. There are many well-know algorithms to analyse data, like clustering and machine learning.

1. LECTURE L18
BIG DATA AND ANALYTICS

2. Data Gathering

3. 1955 1960 1965
Social Security
Calculate Benefits for 15MM
Recipients (62MM Now)
NASA
Calculate Real-Time Orbital
Determination
IRS
Calculate / Store 55MM
Records (126MM Now)
Data Gathering in US 1950+
Source: Mary Meeker Slide Deck 2019

4. 1955 1965 1975
Banks
Process Checks
Data Gathering in US 1950+
Source: Mary Meeker Slide Deck 2019
Telecoms
Optimise Telephone switching
Hospitals
Manage Patient Data
Airlines
Process transaction / data
Insurance
Optimise Insurance Policies
Retail
Track Inventory / Logistics
Credit Cards
Manage Merchant Network
Source: Mary Meeker Slide Deck 2019

5. Big Bangs in Data
2006
Amazon AWS
2007
Apple iPhone
Until now, a sophisticated & scalable data
storage infrastructure has been beyond
the reach of small developers.
— Amazon S3 Launch FAQ, 2006
Why run such a sophisticated operating
system on a mobile device? Well,
because it’s got everything we need.
— Steve Jobs, iPhone Launch, 2007
Source: Mary Meeker Slide Deck 2019

6. Growth of Data
Source: Mary Meeker Slide Deck 2019

7. Decline of Cost
Source: Mary Meeker Slide Deck 2019

8. Increasing Revenues
Source: Mary Meeker Slide Deck 2019

9. Source: Mary Meeker Slide Deck 2019
Where Does the Data Come From?

10. Where Does the Data Come From?
Source: Mary Meeker Slide Deck 2019

11. Where Does the Data Come From?
Source: Mary Meeker Slide Deck 2019

12. Big Data

13. Big Data
With the computer revolution, digital data becomes possible
Over the years, data has grown exponentially
“Big Data” has become a
platform by itself with new
possibilities

14. Global Data is Growing Fast
Data in Digital Universe vs. Data Storage Cost, 2010-2015
Source: Mary Meeker, KPCB

15. Evolution of Data Platform
Source: Mary Meeker, KPCB

16. Data is a New Growth Platform
The
Network
The
Software
The
Infrastructure
The
Data
Large investments in fibre optic & last-mile cable create connectivity
that facilitated the early Internet growth
Optimising the network with software became far more capital
efficient than additional capital expenditure buildouts, ultimately
resulting in the creation of pervasive networks (Siloed DCs -> AWS)
and pervasive software (Siebel -> Salesforce)
Emergence of pervasive software created the need to optimise the
performance of the network and store extraordinary amounts of data
at extremely low prices
Next Big Wave: Leveraging this unlimited connectivity and storage to
collect / aggregate / correlate / interpret all of this data to improve
people’s live and enable enterprises to operate more efficiently

17. Data Generators
Source: Mary Meeker, KPCB

18. “Data is moving from something you use
outside the workstream to becoming a part of
the business app itself.”
— Frank Bien, CEO of Looker

19. Improve people’s live and enable
enterprises to operate more efficiently

20. Big Data Examples

21. Big Data Examples
Macy's Inc. and real-time pricing
The retailer adjusts pricing in near-real time for 73 million
items, based on demand and inventory.
Source:Ten big data case studies in a nutshell

22. Big Data Examples
Tipp24 AG, a platform for placing bets
The company uses software to analyse billions of
transactions and hundreds of customer attributes, and to
develop predictive models that target customers and
personalise marketing messages on the fly.
Source:Ten big data case studies in a nutshell

23. Big Data Examples
Wal-Mart Stores Inc. and search
The mega-retailer's latest search engine for Walmart.com
includes semantic data. A platform that was designed in-
house, relies on text analysis, machine learning and even
synonym mining to produce relevant search results.
Wal-Mart says adding semantic search has improved
online shoppers completing a purchase by 10% to 15%.
Source:Ten big data case studies in a nutshell

24. Big Data Examples
PredPol Inc. and repurposing
The Los Angeles and Santa Cruz police departments, a
team of educators and a company called PredPol have
taken an algorithm used to predict earthquakes, tweaked it
and started feeding it crime data.
The software can predict where crimes are likely to occur
down to 500 square feet. In LA, there's been a 33%
reduction in burglaries and 21% reduction in violent crimes
in areas where the software is being used.
Source:Ten big data case studies in a nutshell

25. Big Data Examples
American Express and business intelligence
AmEx started looking for indicators that could really
predict loyalty and developed sophisticated predictive
models to analyse historical transactions and 115 variables
to forecast potential churn
The company believes it can now identify 24% of Australian
accounts that will close within the next four months
Source:Ten big data case studies in a nutshell

26. Big Data Examples
A Bank and IBM
A large US bank uses IBM machine learning technologies
to analyse credit card transactions.
Using machine learning and stream computing to detect financial fraud

27. TEDxUofM - Jameson Toole - Big Data for Tomorrow

29. What is Big Data?

30. What is Big Data?
Big data is high-volume, high-velocity and/or high-variety
information assets that demand cost-effective, innovative
forms of information processing that enable enhanced
insight, decision making, and process automation.
Gartner

31. What is Big Data?
Big data refers to a process that is used when traditional
data mining and handling techniques cannot uncover the
insights and meaning of the underlying data. Data that is
unstructured or time sensitive or simply very large cannot
be processed by relational database engines. This type of
data requires a different processing approach called big
data, which uses massive parallelism on readily-available
hardware.
Techopedia

32. “Big data is the oil of the 21st century and
analytics is the combustion engine.”
—Peter Sondergaard, Gartner Research
What is Big Data?

33. How do you measure numbers at large scale?
What is Big Data?

34. What is a Yottabyte?

35. 1.000.000.000.000.000.000.000.000
What is a Yottabyte?

36. Byte: one rice
David Wellman: What is Big Data?
What is Big Data?

37. Byte: one rice
Kilobyte: handful of rice
David Wellman: What is Big Data?
What is Big Data?

38. Byte: one rice
Kilobyte: handful of rice
Megabyte: Big pot of rice
David Wellman: What is Big Data?
What is Big Data?

39. Byte: one rice
Kilobyte: handful of rice
Megabyte: Big pot of rice
Gigabyte: Truck full of rice
David Wellman: What is Big Data?
What is Big Data?

40. Byte: one rice
Kilobyte: handful of rice
Megabyte: Big pot of rice
Gigabyte: Truck full of rice
Terabyte: Containership full of rice
David Wellman: What is Big Data?
What is Big Data?

41. Byte: one rice
Kilobyte: handful of rice
Megabyte: Big pot of rice
Gigabyte: Truck full of rice
Terabyte: Containership full of rice
Petabyte: Covers Manhattan
David Wellman: What is Big Data?
What is Big Data?

42. Byte: one rice
Kilobyte: handful of rice
Megabyte: Big pot of rice
Gigabyte: Truck full of rice
Terabyte: Containership full of rice
Petabyte: Covers Manhattan
Exabyte: Covers the west coast of US
David Wellman: What is Big Data?
What is Big Data?

43. Byte: one rice
Kilobyte: handful of rice
Megabyte: Big pot of rice
Gigabyte: Truck full of rice
Terabyte: Containership full of rice
Petabyte: Covers Manhattan
Exabyte: Covers the west coast of US
Zettabyte: Fills the Pacific Ocean
David Wellman: What is Big Data?
What is Big Data?

44. Byte: one rice
Kilobyte: handful of rice
Megabyte: Big pot of rice
Gigabyte: Truck full of rice
Terabyte: Containership full of rice
Petabyte: Covers Manhattan
Exabyte: Covers the west coast of US
Zettabyte: Fills the Pacific
Yottabyte: Earth size riceball
David Wellman: What is Big Data?
What is Big Data?

45. Byte: one rice
Kilobyte: handful of rice
Megabyte: Big pot of rice
Gigabyte: Truck full of rice
Terabyte: Containership full of rice
Petabyte: Covers Manhattan
Exabyte: Covers the west coast of US
Zettabyte: Fills the Pacific
Yottabyte: Earth size riceball
David Wellman: What is Big Data?
Big Data
Internet
Computers
Early computers
What is Big Data?

46. Big Data is not just about the size of
the data, it’s about the value within
the data
This value can be used for marketing,
businesses optimisation, getting
insights, improving health, security
etc.
What is Big Data?

47. Data Analytics

48. Why Big Data Analytics?
Understand the data the company has
Process data to see patterns, corrections and
information that can be used to make better
decisions
Obtain insights that are otherwise not known

49. Data Analytics
TRADITIONAL APPROACH
Structured and Repeatable Analyses
BIG DATA APPROACH
Iternative and Exploratory Analyses
Business users
Business users
Determine what
questions to ask
IT
Structures the data
to answer the
question
IT
Delivers a platform
to enable creative
discovery
Explores what
questions could be
asked

50. Tools for Data Analytics
NoSQL databases: MongoDB, Cassandra, Hbase, Hypertable
Storage: S3, Hadoop Distributed File System
Servers: EC2, Google App Engine, Heroku
MapReduce: Hadoop, Hive, Pig, Cascading, S4, MapR
Processing: R, Yahoo! Pipes, Solr/Lucene, BigSheets,

51. Two Types of Data Analysis Problems
Supervised Learning: Learn from data but we have labels
for all the data we’ve seen so far
Example: Determining Spam Emails
Learn from data but we don’t have any
labels
Example: Grouping Emails, AlphaZero
Unsupervised Learning:
Learning is about discovering hidden patterns in data

52. Clustering
One of the oldest problems in unsupervised data analysis
In clustering the goal is to group data according to similarity
Algorithms such as K-means are used for clustering

53. For each artefact found,
the location to N and E
from the Marker is
recorded
That is a Data Set
Before the dig, a historian
has said that three families
lived in the location
Clustering

54. Similar: close in physical
distance
You assign each data point
to one and only one group
The groups are called
clusters
Clustering

55. Clustering is the unsupervised learning problem where
you take your data and assign each data point to exactly
one group, or cluster
Uses unlabelled data
Clustering

56. We may have collection data but we don’t know what to
do with it
We might want to explore the data without a particular
end goal in mind
Perhaps the data will suggest interesting avenues for
further analysis
In this case, we say that we're performing exploratory
data analysis
Clustering

57. Exploratory data analysis
We don’t know what we are looking for
Data point = colour of pixel and location of pixel
Dissimilarity is the distance in colour

58. In some cases
labelling is too
expensive
For example,
news change
every day and
there are too
much of them
Exploratory data analysis

59. Using Big Data to Influence People

60. Alexander Nix, CEO Cambridge Analytica
Ted Cruz campaign for US Republican President

62. Data Analysis as a Platform
THEN NOW
Complex tools operated by Data Analysts
Chaos of data silos accross the company
Real-time data analytics platform like Looker

63. Customer Data as a Platform
Difficult to customise,
lack of automated
customer insights
Real-time Intelligent that
automatically tracks and analysis
interaction with customer
THEN NOW

64. Mapping Data as a Platform
Difficult and expensive to collect data
Limited in-app digital map usage
Mapping platforms like Mapbox
THEN NOW

65. Cloud Data Monitoring as a Platform
Expensive and clunky point solution
Lengthy implementation cycles
Only used by System Administrators
Cloud monitoring platforms like
Datadog
THEN NOW

66. Next
Lecture L19 Network Platforms