1. Cloud

2. The big data pipeline
How customers are using the pipeline
The big data eco-system on the cloud

3. Generation
Collect
Store
Collaboration & sharing
Analysis and Computation

4. Generation
Collect
Store
Collaboration & sharing
Analysis and Computation
lower cost,
increased
throughput

5. Generation
Collect
Store
Collaboration & sharing
Analysis and Computation
lower cost,
increased
throughput
constraint

6. Generated data
Available for analysis
Data volume
Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011
IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares

7. Very high barrier to
turning data into
information…

8. Very high barrier to
turning data into
information.
Infrastructure capacity
Technical Skills
Questions to ask
Cheap experimentation

9. Amazon Web Services Cloud

10. Elastic and highly scalable
No upfront capital expense
Only pay for what you use
+
+
Available on-demand
+
=
Remove
constraints

11. Remove constraints = More experimentation
More experimentation = More innovation
More Innovation = Competitive edge

12. Amazon Web Services
Removes constraints
Focus on your data
Leave undifferentiated heavy lifting to us

13. big data

14. Bankinter uses HPC on AWS for Monte Carlo
Simulation
“Bankinter uses AWS as an
integral part of our credit-
risk simulation application;
We need to perform at
least 5,000,000 simulations
to get realistic results”
Credit
Data
Average simulation
time went from 23 hours to 20 minutes

15. Challenge:
Learn about customer based on
what they do, rather than what
they say (i.e., data exhaust);
virtually unlimited data
Solution:
Always-on cluster continually
processes new financial data
and stores results in S3.
Collaborative filtering used to
provide recommendations and
ad-hoc queries performed
using Hive.

16. For illustrative purposes only.

17. S&P Capital IQ
Microsoft
SQL Server
Amazon S3:
• Companies You May
Be Interested In
Amazon S3:
• Clicks
• Key Developments
• Company Profiles
Amazon Elastic Map-Reduce:
• Compute User Selectivity
• Compute Key Developments
• Join & Score

19. Challenge:
Volatile weather is deadly to crops like grapes and tomatoes
Solution:
Built a predictive model based on freely available data—60 years of
crop data, 14 TBs of soil data, and one million government Doppler
radar points. 50 hadoop clusters process new data as it comes into S3
each day, continuously updating the model.
150B Soil
Observations
3M Daily
Weather
Measurements
850K Precision
Rainfall Grids
Tracked

20. Simulations Each Month
• Per Simulation:
• 10K Unique Scenarios Generated
• 5 Trillion Datapoints
• 5-6k Node Hadoop Cluster

24. AWS
Import/Export
Corporate
data center
Amazon
Elastic
MapReduce
Amazon
Simple
Storage
Service (S3)
BI Users
Clickstream data
from 500+
websites and VoD
platform

25. More than 25 Million Streaming Members
50 Billion Events Per Day
30 Million plays every day
2 billion hours of video in 3
months
4 million ratings per day
3 million searches
Device location , time ,
day, week etc.
Social data

26. 10 TB of streaming data per day

27. Data consumed in multiple ways
S3
EMR
Prod Cluster
(EMR)
Recommen
dation
Engine
Ad-hoc
Analysis
Personalization

33. Amazon Dynamodb

34. “Who buys video games?”

35. 3.5 billion records
13 TB of click stream logs
71 million unique cookies
Per day:

37. 500% return on ad spend
17,000% reduction in
procurement time
Results:

38. “Who is using our
service?”

39. Identified early mobile usage
Invested heavily in mobile
development
Finding signal in the noise of logs
9,432,061 unique mobile devices
used the Yelp mobile app.

43. Every day is crucial and costly

44. Challenge: To run a virtual screen with a higher
accuracy algorithm & 21 million compounds

46. Metric Count
Compute Hours of
Work
109,927 hours
Compute Days of
Work
4,580 days
Compute Years of
Work
12.55 years
Ligand Count ~21 million ligands
Using Cycle Computing and Amazon
Web Services

47. 3 Hours
for $4828.85/hr

49. Relational Database Service
Fully managed database
(MySQL, Oracle, MSSQL)
DynamoDB
NoSQL, Schemaless,
Provisioned throughput
database
S3
Object datastore up to 5TB
per object
99.999999999% durability

50. Map-Reduce engine
Hadoop-as-a-service
Massively parallel
Cost effective AWS wrapper
Amazon Elastic MapReduce

51. Amazon
Redshift
data warehouse service
petabyte-scale
fast and fully managed

52. RDBMS
Redshift
OLTP
ERP
Reporting
and BI

54. +
Source: http://nerds.airbnb.com/redshift-performance-cost
Table Size Query type Hive Redshift
3 billion
rows
Simple range
query
1680
seconds (28
min)
360 seconds
(6 min)
1 million
rows
2 complex
joins
182 seconds 8 seconds
$13.60/hour on Redshift versus $57/hour on
HIVE

55. Generation
Collect
Store
Collaboration & sharing
Analysis and Computation

56. Thank you! aws.amazon.com/big-data
May 14st, Kowloonbay International Trade
& Exhibition Centre (KITEC), Hong Kong
One day Free training
Walk through of services
http://aws.amazon.com/apac/awsday/hk/