AWS Cloud Kata 2014 | Jakarta - 2-3 Big Data

1. Jakarta
Big Data on AWS
Markku Lepistö
Principal Technology Evangelist
@markkulepisto

2. Does this Data make me look big?

3. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

5. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

6. Getting your Data into AWS
Amazon S3
Corporate(Data(
Center(
• Console Upload
• FTP
• AWS Import Export
• S3 API
• Direct Connect
• Storage Gateway
• 3rd Party Commercial Apps
• Tsunami UDP
1

7. Services: Storage: Amazon S3
Deployment(&(Administra=on(
App(Services(
Compute( Storage(
Database(
Networking(
AWS(Global(Infrastructure(
Amazon Simple Storage Service (S3)
• Unlimited storage of objects of any type
• 99.999999999% durability, replicated across multiple facilities
• Cost effective storage, US$0.03/GB Month
• Granular access control and permissions over objects
• Encryption at rest using AES 256bit server side encryption
• Encryption in transit using HTTPS protocol
• High performance throughput supporting parallelized
upload or download
• Import or export data via physical device handling service
• Data remains in geographic location chosen

8. Write directly to a data source
Your%applica+on% Amazon S3
DynamoDB%
Any%other%data%
store%
Amazon S3
Amazon%EC2%%
2

9. Services: Database: Amazon DynamoDB
" Zero Admin NoSQL Service
" Unlimited Storage
" Provisioned Throughput
" Consistent <10ms response
" Durable on SSD
Compute( Storage(
Database(
Networking(
AWS(Global(Infrastructure(

10. Scale or Autoscale DynamoDB Table Throughput

11. Queue, pre-process and then write to data source
Amazon%Simple%
Queue%Service%
(SQS)%
Amazon S3
DynamoDB%
Any%other%data%
store%
3

16. Aggregate and write to data source
Flume running
on EC2
Amazon S3
HDFS
Any%other%data%
store%
4

17. Choose depending upon design
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%

18. S3 as a single source of truth
S3
Courtesy http://techblog.netflix.com/2013/01/hadoop-platform-as-service-in-cloud.html

19. Generation
Collection storage
Analytics computation
Collaboration sharing

20. Hadoop based Analysis
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR

21. What is Amazon Elastic MapReduce (EMR)?
EMR is Hadoop in the Cloud!

22. How does EMR work ?
EMR%Cluster
S3
Put the data
into S3
Choose: Hadoop distribution, # of
nodes, types of nodes, custom
configs, Hive/Pig/etc.
Get the output from
S3
Launch the cluster using the
EMR console, CLI, SDK, or
APIs
You can also store
everything in HDFS

23. What can you run on EMR…
S3
EMR Cluster

24. SQL based processing
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR
Amazon
Redshift
Pre-processing
framework
Petabyte scale
Columnar Data -
warehouse

25. Services: Database: Amazon Redshift
Amazon Redshift
• Easily and rapidly analyze petabytes of data
• Fully managed data warehouse service
• Automated deployment and administration
• 1/10th the cost of traditional data warehouses
• $1000 / Terabyte / year
• Compatible with popular BI tools
Deployment((Administra=on(
App(Services(
Compute( Storage(
Database(
Networking(
AWS(Global(Infrastructure(

26. Your choice of BI Tools on the cloud
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR
Amazon
Redshift
Pre-processing
framework

27. Generation
Collection storage
Analytics computation
Collaboration sharing

28. Collaboration and Sharing insights
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR
Amazon
Redshift

29. Sharing results and visualizations at scale
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR
Amazon
Redshift
Visualization tools
Web App Server

30. Rinse and Repeat every day or hour

31. Rinse and Repeat
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR
Amazon
Redshift
Business
Intelligence Tools
Visualization tools
GIS tools
Business
Intelligence Tools
GIS tools on
hadoop
Amazon data pipeline

32. The complete architecture
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR
Amazon
Redshift
Business
Intelligence Tools
Visualization tools
GIS tools
Business
Intelligence Tools
GIS tools on
hadoop
Amazon data pipeline

33. No it isn’t !

34. What about Real-Time?

35. nopeampi data on parempi data

36. is the exception rate
is the ad click-through
topics are trending
inventory remains
queries are slow
are the high scores
What right now?

37. HAPPENING NOW!
real-time == stream analytics

38. Ingest data streams
Store durably
Distribute
Scale out
Process as packets flow in

40. Realtime Analytics in the Cloud
Amazon%Kinesis%
Streaming*Data*Service*

41. Kinesis architecture
AZ AZ AZ
Durable, highly consistent storage replicates data
across three data centers (availability zones)
Amazon Web Services
Aggregate and
archive to S3
Millions of
sources producing
100s of terabytes
per hour
Front
End
Authentication
Authorization
Ordered stream
of events supports
multiple readers
Real-time
dashboards
and alarms
Machine learning
algorithms or
sliding window
analytics
Aggregate analysis
in Hadoop or a
data warehouse
Inexpensive: $0.028 per million puts

43. InBgame(
ac=vity(
Kinesis:(RealB=me(data(stream(of(inBgame(ac=vity(
Amazon(
Kinesis(
Clash(of(Clans(

44. KinesisBenabled(apps(on(EC2(
InBgame(
ac=vity(
Kinesis:(RealB=me(data(stream(of(inBgame(ac=vity(
Mul=ple(Kinesis(applica=ons:(Dashboards,(analy=cs(and(storage(
Clash(of(Clans(
RealB=me(clickstream(
processing(app(
Amazon(
Kinesis(

45. Kinesis:(RealB=me(data(stream(of(inBgame(ac=vity(
Mul=ple(Kinesis(applica=ons:(Dashboards,(analy=cs(and(storage(
S3(and(Glacier:(Data(storage(and(long(term(archival(
S3( Aggregate(
sta=s=cs(
InBgame(
ac=vity(
EC2:(InBgame(
engagement(
trends(
dashboard(
Clash(of(Clans(
KinesisBenabled(apps(on(EC2(
RealB=me(clickstream(
processing(app(
Amazon(
Kinesis(

46. BusinessBintelligence(
user(
Kinesis:(RealB=me(data(stream(of(inBgame(ac=vity(
Mul=ple(Kinesis(applica=ons:(Dashboards,(analy=cs(and(storage(
EC2:(InBgame(
engagement(
trends(
dashboard(
InBgame(
ac=vity(
S3( Aggregate(
sta=s=cs(
S3(and(Glacier:(Data(storage(and(long(term(archival(
Data(Warehouse:(BI(repor=ng(and(interac=ve(queries(
Clash(of(Clans(
KinesisBenabled(apps(on(EC2(
EC2(
Data(
Warehouse(
RealB=me(clickstream(
processing(app(
Amazon(
Kinesis(

47. Glacier(
Clickstream(
archive(
EC2(
Data(
Warehouse(
Kinesis:(RealB=me(data(stream(of(inBgame(ac=vity(
Mul=ple(Kinesis(applica=ons:(Dashboards,(analy=cs(and(storage(
EC2:(InBgame(
engagement(
trends(
dashboard(
S3(and(Glacier:(Data(storage(and(long(term(archival(
Data(Warehouse:(BI(repor=ng(and(interac=ve(queries(
RealB=me(clickstream(
processing(app(
InBgame(
ac=vity(
S3(
Clash(of(Clans(
Aggregate(
sta=s=cs(
BusinessBintelligence(
user(
KinesisBenabled(apps(on(EC2(
Amazon(
Kinesis(

48. Demo
Sliding Window Analytics ! Live Dashboard
S3 Storage ! Redshift Data Warehouse
Website
Clickstream
logs
Kinesis

49. 台北
6月25日

50. Bonus
Internet of Things

51. Smart(Devices(
Powered(by(the(Cloud(

52. Smart(Devices(
Powered(by(the(Cloud(

53. Smart(Devices(
Powered(by(the(Cloud(

54. Smart(Devices(
Powered(by(the(Cloud(

55. Smart?evices(
Powered(by(the(Cloud(

56. Smart?evices(
Powered(by(the(Cloud(
Arduino%Uno% Raspberry%Pi%
CPU( 20MHz(8bit( 700MHz(32bit(
Memory( 2(KB( 512(MB(
Storage( 32(KB( SD(card(

57. Smart(Devices(
Powered(by(the(Cloud(

58. Camera(
Microphone(
Thermometer(
Distance(
GPS(
Gyroscope(
Accelerometer(
Actuator(
Relay(
Motor(
Manipulator(
Pressure( Switch(
Wheel(
Propeller(
Rotor(

59. Challenges(

60. Challenges(
Thousands(–(Millions(of(
Devices(/(Producers(

61. Challenges(
Thousands(–(Millions(of(
Devices(/(Producers(
Thousands(–(Millions(of(
Users(/(Consumers(

62. Distributed(
Thousands(–(Millions(of(
Devices(/(Producers(
Thousands(–(Millions(of(
Users(/(Consumers(

63. At(scale(
Thousands(–(Millions(of(
Devices(/(Producers(
Thousands(–(Millions(of(
Users(/(Consumers(

64. Smart(Devices(
Powered(by(the(Cloud(

65. Smart(Devices(
Unlimited(Storage(–(Memory(
Powered(by(the(Cloud(
((
Unlimited(Compute(–(Logic(

67. Camera(
Microphone(
Thermometer(
Distance(
GPS(
Gyroscope( Accelerometer(
Actuator(
Relay(
Motor(
Manipulator(
Pressure( Switch(
Wheel(
Propeller(
Rotor(

68. Smart(Devices(
Powered(by(the(Cloud(

70. Dropcam(is(the(biggest(inbound(video(service(on(the(Web((
• More(data(uploaded(per(
minute(than(YouTube((
• Petabytes(of(data(
processed(every(month(
• Billions(of(mo=on(events(
detected(

72. 73(

74. The AWS Big Data Portfolio
COLLECT | STORE | ANALYZE | SHARE
Direct Connect S3
Import Export
S3 EC2
DynamoDB Redshift
Glacier
EMR
Data Pipeline
Kinesis
CloudFront

75. (((((((((((((((AWS(CBSDK(Experimental(
for((
ArduinoBstyle(IoT(Devices(

76. CBSDK(–(Na=ve(AWS(Libraries,(
Direct(Access(to(AWS(Services(from(Devices(

77. Demo(

78. Arduino(Yún(
%

79. Raspberry(Pi(

80. Spark(Core(

81. AcceleB
rometer(
MQTT(
Mosqui_o(MQTT(Broker(
MQTTBKinesis(Bridge(
AWS(SDK(
Amazon(Kinesis(
RealB=me(Streaming(
Data(Service(
AWS(APIs(
AWS(Elas=c(
Beanstalk(
Dashboard(

82. Amazon(SNS(
Alert(No=fica=on(
(5g(
Mobile(Push(
Spark(API(
Sound(Alarm(
(6g((
AWS(Elas=c(
Beanstalk(
Dashboard(

83. Demo(

84. AWS Cloud Kata for Start-Ups and Developers
台北
6月25日