This document discusses building scalable big data solutions. It begins with an introduction from Bob Rogers, Chief Data Scientist for Intel. The next section features a presentation from Durga Nemani of AOL on their hybrid approach to big data. Some key aspects of AOL's architecture include separating compute and storage across Amazon S3 and EMR, using transient clusters of varying sizes, and optimizing for cost by using spot instances. The presentation provides data and insights such as processing 150 TB of compressed data per day across 350 EMR clusters. It breaks down AWS costs, with storage being the largest recurring fee. Best practices are suggested around resource tagging, infrastructure as code, security, and scaling. The presentation concludes by discussing next steps

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Bob Rogers, PhD Chief Data Scientist for Big Data Solutions, Intel
Durga Nemani, System Architect AOL Inc.
October 2015
Building Scalable Big Data
Solutions
BDT210

2. Building Scalable Big Data Solutions
October 2015
Bob Rogers, PhD
Chief Data Scientist for Big Data Solutions
Intel

3. @scientistBob 3
About me

4. @scientistBob
What does Big Data have to do with Intel?
Trusted Analytics Platform

5. @scientistBob 5
Intel contributions to Apache Hadoop
Encryption
Intel® AES-NI

6. @scientistBob 6
Use case:
Assemble an accurate patient problem list
Why?
• To improve patient outcomes
KPI
• False negatives in problem list

7. @scientistBob 7
What does a patient look like to a data scientist?

8. @scientistBob
8
My first enterprise data hub

9. @scientistBob
0-25 %
25-50 %
50-75%
75-100 %
Poll: What percent of the key clinical data to you think is missing from
the problem list?
?

10. @scientistBob
>63%
Missing
Poll: What percent of the key clinical data to you think is missing from
the problem list?

11. @scientistBob
Real patient example
Coded
Data
Free Text
Scanned
Document
s
Other
Data Silos

12. @scientistBob
Missing information

13. @scientistBob 13
What did we learn?
• Start with what you know
• Leverage existing
technologies
• Use simple tools
• Measure your results

14. @scientistBob
Powerful Big Data analytics reveal the truth about your…
…customers
…products
…ecosystem
…opportunities
14

15. Thank you
bob.rogers@intel.com
@scientistBob
15

16. BuildingScalableBigData
Solutions
Durga Nemani – AOL Inc.

17. BACKGROUND&
ARCHITECTURE

18. HYBRID

19. TheThree Vs
• Volume
• Multiple Terabytes per day
• Variety
• Delimited, Avro, JSON
• Velocity
• Hourly, Batch

20. Workload Management
• “One size fits all” model does not work.
• Specific infrastructure tuned to needs and requirements
• Variety of EMR clusters as per Data need
2
0
Workloads with significant
diversity of needs
Resources with lowest
common denominator
Resources for
workloads with significant
diversity of needs

21. S3
EMR
EMR
EMR
EMR

22. JSON
EC2EMRS3
Apache HiveApache PigApache Hadoop
Open Source Data
Formats
AWS Services
Open Source
Technologies
Avro Parquet

23. UNIQUE
FEATURES&
ADVANTAGES

24. Separation of Computeand Storage

25. SEE,SPOT,SQUEEZE
• Just enough spot instances to finish the job in 59 minutes.

26. KeyFeatures
• Separation of Compute and Storage: Amazon S3 and Amazon EMR
• Transient Clusters: No permanent cluster. Different size clusters for
different datasets
• Separation of duties: Independent jobs for Processing,
Extracting, loading and monitoring.
• Parallelism: Process the smallest chunk of data possible in
parallel to reduce dependencies
• Scalability: Hundreds of Amazon EMR clusters in multiple
regions and Availability Zones
• Cost optimized: All Spot instances. Launch in Availability Zone
with lowest spot prices.

27. DATA&
INSIGHTS

28. CLOUD Facts
2
8
Total Compressed
Amazon S3 Data Size
150 TB
Uncompressed
RAW Data/Day
2-3 TB
Amazon EMR
Clusters/Day
350
Amazon S3 Data
Retention Period
13-24 Months

29. 150
24,000
Restatement Use Case
Terabytes raw
2
9
10
Availability Zone
550
EMR Clusters EC2 Instances

30. AWS COSTBREAKOUT
44%
40%
16%
3
0
** Storage cost is recurring every month at 2.85$/100 GB
EC2 Cost
EMR Fee
S3 Cost

31. BestPractices&
Suggestions

32. Tag all resources
Infrastructure as
Code
Command Line Interface
JSON as configuration files
AWS Identity and
Access Management
(IAM) roles and policies
Use of application ID
Enable CloudTrail
S3 lifecycle
management
S3 versioning
Separate code/data/logs buckets
Keyless EMR
clusters
Hybrid model
Enable debugging
Create multiple CLI profiles
Multi-factor authentication
CloudWatch billing alarms
EC2 Spot
instances
SNS notifications for failures
Loosely coupled Apps
Scale horizontally

33. NextSteps

34. 3
4
Database on cloud
• Database on AWS
• Options: Amazon RDS, Amazon Redshift, or others using
Amazon EC2
Event-driven design
• Kick off code based on events
• Run downstream processes as soon as upstream completes
• Options: AWS Lambda, Amazon SQS, Amazon SWF or AWS
Data Pipeline
Data analytics
• Implement massive parallel processing technologies
• Options: Spark, Impala or Presto
DevOPS on cloud
• Rapidly and automatically deploy new code
• Continuous Integration/Continuous Deployment
• Options: AWS CodeDeploy, AWS CodeCommit, or AWS
CodePipeline

35. Q&A

36. THANKYOU
Recommended session:
BDT208 - A Technical Introduction to
Amazon Elastic MapReduce
Thursday, Oct 8, 12:15 PM - 1:15 PM
– Titian 2201B

37. Remember to complete your
evaluations!