In this webinar AWS Technical Evangelist, Ian Massingham, discusses the role that AWS services can play in helping you to derive value from your data, from stream processing with Amazon Kinesis, techniques for managing ingest of large data sets, through to processing data with Amazon Elastic MapReduce (EMR) and its ecosystem of tools and running large scale data warehouses on AWS with Redshift. View the recording: http://youtu.be/7bkqopn19WY

1. Journey through the Cloud:
Big Data
Ian Massingham – Technical Evangelist
@IanMmmm

2. Journey through the cloud
Common use cases & stepping stones into the AWS cloud
Learning from customer journeys
Best practices to bootstrap your projects

3. Big Data on AWS
Collect and store Big Data in the AWS Cloud
Meet the challenge of the increasing volume, variety, and velocity of data
Reduce costs, scale to meet demand, and increase the speed of innovation
Make use of solutions for every stage of the big data lifecycle

4. Agenda
Collecting Big Data in the AWS Cloud
Real-time Streaming and Analysis
Big Data Cloud Storage Solutions
Analytics with Hadoop with Amazon EMR
Case Studies & Useful Resources

5. Why Run Your Big Data
Applications on AWS?

6. We are constantly producing more data

7. From all types of industries

8. From a diverse range of sources

9. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

10. Lower cost,
higher throughput
GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

11. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Highly
constrained
Lower cost,
higher throughput

12. + ELASTIC AND HIGHLY SCALABLE
+ NO UPFRONT CAPITAL EXPENSE
+ ONLY PAY FOR WHAT YOU USE
+ AVAILABLE ON-DEMAND
= REMOVE CONSTRAINTS

15. SIMULATING
205,000 COMPOUNDS!

17. 18 HOURS
FOR $1833.33/hr!

18. Instead of
$68+ MILLIONS
in infrastructure!

19. Collecting Big Data in
the AWS Cloud

20. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

21. AWS Import / Export
AWS Direct Connect
Amazon Storage Gateway
GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

22. Getting data into the AWS Cloud
AWS Direct
Connect
Direct connect, import/export and storage gateway
Dedicated bandwidth between
you site and AWS
Amazon Storage
Gateway
Gateway-Stored Volumes
Shrink-wrapped gateway for volume
synchronization
AWS Import/Export
Physical transfer of media into
and out of AWS

23. Inbound data transfer is free
Multipart upload to S3
Physical media via AWS Import/Export
AWS Direct Connect

24. Real-time Streaming
and Analysis

25. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

26. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
BATCH
PROCESSING

27. STREAM
PROCESSING
GENERATE ➔ ➔ SHARE!

28. Hourly server logs: how your
systems went wrong an hour ago
Weekly / Monthly Bill: What you
spent this past billing cycle
Daily customer report from your
website: tells you what deal or ad
to try next time
Daily fraud reports: tells you if there
was fraud yesterday
Daily business reports: tells me
how customers used my services
yesterday
Real-time metrics: what just went
wrong now
Real-time spending alerts/caps:
guaranteeing you can’t overspend
Real-time analysis: what to offer
the current customer now
Real-time detection: blocks
fraudulent use now
Fast ETL into Amazon Redshift:
how are customers using my
services now

29. AMAZON KINESIS
REAL-TIME DATA STREAM PROCESSING!
aws.amazon.com/kinesis

30. AMAZON KINESIS ARCHITECTURE"
Millions of sources
producing 100s of
TB per hour!
Front
End
Authentication!
Authorization!
AZ!AZ!AZ!
Durable, highly consistent storage replicates
data across three centers (availability zones)!
Amazon Web Services!
Inexpensive: $0.028 per million puts!
Aggregate and
archive to S3!
Real-time
dashboards
and alarms!
Machine
learning
algorithms !
Aggregate analysis
in Hadoop or a
data warehouse!
Ordered stream of events
supporting multiple readers!

31. Big Data Cloud
Storage Solutions

32. Fundamental Storage Options
Elastic Block Store, S3 and Glacier
Simple Storage Service
Highly scalable object storage
1 byte to 5TB in size
99.999999999% durability
Elastic Block Store
High performance block storage device
1GB to 1TB in size
Mount as drives to instances with
snapshot/cloning functionalities
Glacier
Long term object archive
Extremely low cost per gigabyte
99.999999999% durability

33. Fundamental Storage Options
Elastic Block Store, S3 and Glacier
Very fast
‘instance’
disks
Slow, rare
access
Simple Storage Service
Highly scalable object storage
1 byte to 5TB in size
99.999999999% durability
Fast web object
storage
Elastic Block Store
High performance block storage device
1GB to 1TB in size
Mount as drives to instances with
snapshot/cloning functionalities
Glacier
Long term object archive
Extremely low cost per gigabyte
99.999999999% durability

34. Fundamental Storage Options
Elastic Block Store, S3 and Glacier
Elastic Block Store
High performance block storage
device
1GB to 1TB in size
Mount as drives to instances with
snapshot/cloning functionalities
Glacier
Long term object archive
Extremely low cost per gigabyte
99.999999999% durability
IMAGE
Amazon S3
Simple Storage Service
Highly scalable
data storage in-the-cloud
Programmatic access
via web services API
Is a Web Store
Not a file system
Fast, highly available
Durable
Economical
Paradigm Object store
Performance Very fast
Redundancy Across data centers
Security Public Key / Private Key
Pricing
$0.03/GB/month stored
(eu-west-1)
Access from
the Net?
Yes
Typical use
case
Write once, read many
Simple Storage Service
Highly scalable object storage
1 byte to 5TB in size
99.999999999% durability

35. Analytics with Hadoop
on Amazon EC2

37. AMAZON ELASTIC
MAPREDUCE
HADOOP AS A SERVICE!

38. • SPLITS DATA INTO PIECES
• LETS PROCESSING OCCUR
• GATHERS THE RESULTS!

39. Very large
click log
(e.g TBs)

40. Lots of actions by
John Smith
Very large
click log
(e.g TBs)

41. Lots of actions by
John Smith
Split the log
into many
small pieces
Very large
click log
(e.g TBs)

42. Lots of actions by
John Smith
Split the log
into many
small pieces
Process in an
EMR cluster
Very large
click log
(e.g TBs)

43. Lots of actions by
John Smith
Split the log
into many
small pieces
Process in an
EMR cluster
Aggregate
the results
from all the
nodes
Very large
click log
(e.g TBs)

44. Lots of actions by
John Smith
Split the log
into many
small pieces
Process in an
EMR cluster
Aggregate
the results
from all the
nodes
Very large
click log
(e.g TBs)
What
John
Smith
did

45. What
John
Smith
did
Insight in a fraction of the time
Very large
click log
(e.g TBs)

46. Analytics Data management languages/engines
HDFS
Pig
Amazon
Redshift
AWS Data Pipeline
Amazon
Kinesis
Amazon
S3
Amazon
DynamoDB
Amazon
RDS
Amazon
EMR
Data
Sources

47. ODBC and JDBC drivers now for Amazon EMR
Amazon EMR now ships with ODBC and JDBC drivers for Hive, Impala & HBase
Now easier to use popular BI tools like:
Microsoft Excel, Tableau, MicroStrategy, and QlikView

48. How customers are
using AWS for Big Data

49. aws.amazon.com/solutions/case-studies/big-data/!

52. Foursquare…
33 million users
1.3 million businesses
…generates a lot of Data
3.5 billion check-ins
15M+ venues,
Terabytes of log data

53. Uses EMR for
Evaluation of new features
Machine learning
Exploratory analysis
Daily customer usage reporting
Long-term trend analysis

54. Benefits of Amazon EMR
Ease-of-Use
“We have decreased the processing time for urgent data-analysis”
Flexibility
To deal with changing requirements & dynamically expand reporting clusters
Costs
“We have reduced our analytics costs by over 50%”

55. Gorilla Coffee
Gray's Papaya
Amorino
When do people go to a place?
Thursday
Friday
Saturday
Sunday

58. AWS Services
for Big Data

59. Amazon
S3
Amazon
Kinesis
Amazon
DynamoDB
Amazon
Redshift
Amazon
Elastic
MapReduce

60. Store anything
Object storage
Scalable
99.999999999% durability
Amazon
S3
aws.amazon.com/s3

61. Real-time processing
High throughput; elastic
Easy to use
EMR, S3, Redshift, DynamoDB Integrations
Amazon
Kinesis
aws.amazon.com/kinesis

62. NoSQL Database
Seamless scalability
Zero admin
Single digit millisecond latency
Amazon
DynamoDB
aws.amazon.com/dynamodb

63. Relational data warehouse
Massively parallel
Petabyte scale
Fully managed
$1,000/TB/Year
Amazon
Redshift
aws.amazon.com/redshift

64. Hadoop/HDFS clusters
Hive, Pig, Impala, Hbase
Easy to use; fully managed
On-demand and spot pricing
Integration with S3, DynamoDB & Kinesis
Amazon
Elastic
MapReduce
aws.amazon.com/elasticmapreduce

65. Where to go
to learn more

66. Resources
aws.amazon.com/bigdata
aws.amazon.com/importexport
aws.amazon.com/directconnect
aws.amazon.com/kinesis
aws.amazon.com/elasticmapreduce

67. Big Data Blog
blogs.aws.amazon.com/bigdata

68. AWS Big Data Training
Fundamentals Course
http://aws.amazon.com/training/course-descriptions/bigdata-fundamentals/
This is a free, online training course and is intended for individuals who are new to big data concepts,
including solutions architects, data scientists, and data analysts.
Big Data on AWS
http://aws.amazon.com/training/course-descriptions/bigdata/
The instructor led Big Data on AWS course introduces you to cloud-based big data solutions and Amazon
Elastic MapReduce (EMR), the AWS big data platform. In this course, we will show you how to use
Amazon EMR to process data using the broad ecosystem of Hadoop tools like Pig and Hive.

69. Webinar Resources
Amazon Elastic MapReduce Masterclass
https://www.brighttalk.com/webcast/9019/103985?channel=CP
Amazon Redshift Masterclass
Coming up on September 23
Register here: http://aws.amazon.com/webinars/emea-masterclass/

70. AWS Training & Certification
Certification
Demonstrate your skills,
knowledge, and expertise
with the AWS platform
aws.amazon.com/certification
Self-Paced Labs
Try products, gain new
skills, and get hands-on
practice working with
AWS technologies
aws.amazon.com/training/
self-paced-labs
Training
Skill up and gain
confidence to design,
develop, deploy and
manage your applications
on AWS
aws.amazon.com/training

71. Ian Massingham – Technical Evangelist
@IanMmmm
@AWS_UKI for local AWS events & news
@AWScloud for Global AWS News and Announcements
©Amazon.com, Inc. and its affiliates. All rights reserved.

72. We typically see customers start by trying our services
Get started now at : aws.amazon.com/getting-started

73. Design your application for the AWS Cloud
More details on the AWS Architecture Center at : aws.amazon.com/architecture