Slides from the online seminar, August 1st 2012.

1. Big Data Analytics
with Amazon Web Services
Dr. Matt Wood
An Online Seminar for Partners. Wednesday 1st August.

2. Hello, and thank you.

3. Big Data Analytics
An introduction

4. Big Data Analytics
An introduction
The story of analytics on AWS

5. Big Data Analytics
An introduction
The story of analytics on AWS
Integrating partners

6. Big Data Analytics
An introduction
The story of analytics on AWS
Integrating partners
Partner success stories

7. 1
INTRODUCING BIG DATA

8. Data for competitive
advantage.

9. Using data
Customer segmentation,
financial modeling,
system analysis,
line-of-sight,
business intelligence.

10. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

11. Cost of data generation
is falling.

12. lower cost,
increased throughput
Generation
Collection & storage
Analytics & computation
Collaboration & sharing

13. Generation
HIGHLY CONSTRAINED
Collection & storage
Analytics & computation
Collaboration & sharing

14. Very high barrier to turning
data into information.

15. Move from a
data generation challenge to
analytics challenge.

16. Enter the Cloud.

17. Remove the constraints.

18. Enable data-driven innovation.

19. Move to a distributed data
approach.

20. Maturation of two things.

21. Software for distributed
storage and analysis
Maturation of two things.

22. Software for distributed
storage and analysis
Maturation of two things.
Infrastructure for distributed
storage and analysis

23. Software
Frameworks for
data-intensive workloads.
Distributed by design.

24. Infrastructure
Platform for
data-intensive workloads.
Distributed by design.

25. Support the
data timeline.

26. Generation
HIGHLY CONSTRAINED
Collection & storage
Analytics & computation
Collaboration & sharing

27. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

28. Lower the
barrier to entry.

29. Accelerate time to market
and increase agility.

30. Enable new business
opportunities.

31. Washington Post
Pinterest
NASA

32. “AWS enables Pfizer to explore
difficult or deep scientific
questions in a timely, scalable
manner and helps us make better
decisions more quickly”
Michael Miller, Pfizer

33. 2
THE STORY OF ANALYTICS

34. EC2
Utility computing.
6 years young.

35. Scale out systems
Embarrassingly parallel problems.
Queue based distribution.
Small, medium and high scale.

39. Cost optimization.
EC2
Utility computing.
6 years young.

40. Achieving economies of scale
100%
Time

41. Achieving economies of scale
100%
Reserved capacity
Time

42. Achieving economies of scale
100%
On-demand
Reserved capacity
Time

43. Achieving economies of scale
UNUSED CAPACITY
100%
On-demand
Reserved capacity
Time

44. Spot Instances
Bid on unused EC2 capacity.
Very large discount.
Perfect for batch runs.
Balance cost and scale.

45. $650 per hour

46. Map/reduce
Pattern for distributed computing.
Software frameworks such as
Hadoop.
Write two functions. Scale up.

47. Map/reduce
Pattern for distributed computing.
Software frameworks such as
Hadoop.
Write two functions. Scale up.
Complex cluster configuration
and management.

48. Amazon Elastic MapReduce
Managed Hadoop clusters.
Easy to provision and monitor.
Write two functions. Scale up.
Optimized for S3 access.

49. S3
Input data
UNDER
THE
i
i
HOOD

50. S3
Input data
Code Elastic
MapReduce
UNDER
THE
i
i
HOOD

51. S3
Input data
Code Elastic Name
MapReduce node
UNDER
THE
i
i
HOOD

52. S3
Input data
Code Elastic Name
MapReduce node
Elastic
cluster
UNDER
THE
i
i
HOOD

53. S3
Input data
Code Elastic Name
MapReduce node
HDFS
Elastic
cluster
UNDER
THE
i
i
HOOD

54. S3
Input data
Code Elastic Name
MapReduce node
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster
UNDER
THE
i
i
HOOD

55. S3
Input data
Code Elastic Name Output
MapReduce node S3 + SimpleDB
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster
UNDER
THE
i
i
HOOD

56. S3
Input data
Output
S3 + SimpleDB
UNDER
THE
i
i
HOOD

71. Performance

72. Performance
Compute performance

73. UNDER
THE
i
i
Cluster Compute HOOD
Intel Xeon E5-2670
10 gig E non-blocking network
60.5 Gb
Placement groupings

74. UNDER
THE
i
i
Cluster Compute HOOD
Intel Xeon E5-2670
10 gig E non-blocking network
60.5 Gb
Placement groupings
+ GPU enabled instances

75. Performance
Compute performance

76. IO performance
Performance
Compute performance

77. NoSQL
Unstructured data storage.

78. DynamoDB
Predictable, consistent performance
Unlimited storage
Single digit millisecond latencies
No schema for unstructured data
Backed on solid state drives

79. ...and SSDs for all.
New Hi1 storage instances.

80. UNDER
THE
i
i
hi1.4xlarge HOOD
2 x 1Tb SSDs
10 GigE network
HVM: 90k IOPS read, 9k to 75k write
PV: 120k IOPS read, 10k to 85k write

81. “The hi1.4xlarge configuration is
about half the system cost for the
same throughput.”
Netflix
http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html

82. EBS
Elastic Block Store

83. Provisioned IOPS
Provision required IO performance

84. Provisioned IOPS
Provision required IO performance
+
EBS-optimized instances
with dedicated throughput

85. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

86. Performance + ease of use

87. 3
PARTNER INTEGRATION

88. Extend platform with
partners

89. Innovate on behalf of
customers

90. Remove undifferentiated
heavy lifting

91. MapR distribution for EMR
Rolled the Amazon Hadoop
optimizations into MapR
Choice for EMR customers
Easy deployment for MapR customers

92. MapR distribution for EMR
Hadoop distribution
Integrated into EMR
NFS and ODBC drivers
High availability and cluster
mirroring

93. Informatica on EMR
Enterprise data toolchain
“Swiss army knife” for data formats
Data integration
Available to all on EMR

94. AWS Marketplace
Karmasphere, Marketshare, Acunu Cassandra,
Metamarkets, Aspera and more.
aws.amazon.com/marketplace

95. 4
PARTNER SUCCESS STORIES

96. Razorfish

97. 3.5 billion records
71MM unique cookies
1.7MM targeted ads
per day

98. 3.5 billion records
71MM unique cookies
1.7MM targeted ads
per day
500% improvement in return on ad spend.

99. Cycle Computing
+ Schrodinger

100. 30k cores, $4200 an hour
(compared to $10+ million)

101. Marketshare
+ Ticketmaster
Optimize live event pricing

102. Reduced developer
infrastructure
management time
by 3 hours a day

103. Thank you!

104. Q&A
matthew@amazon.com
@mza on Twitter