3. Amazon RedshiftAmazon Elastic MapReduce
Amazon Glacier
Amazon
DynamoDB
Amazon Machine
Learning
Amazon Kinesis
Combinational Services for Data analytics
Data WarehouseSemi-structured NoSQL Predictive Models Other AppsStreaming
Amazon Simple
Storage Service
Data Lake Archive
Log Generator
4. Create EC2 instance to generate logs
• AMI -> Public Images -> AMI Name : da-hands-on
• Select the AMI and Click Launch
• Instance Type: t2.medium
• Tag: Name - myname-dev
• Security group with SSH ingress opened
$ aws ec2 create-security-group --group-name andy-ssh-sg --description "open SSH
only" --vpc-id vpc-33d27056
{
"GroupId": "sg-7f3dd918"
}
7. Amazon RedshiftAmazon Elastic MapReduce
Amazon Glacier
Amazon
DynamoDB
Amazon Machine
Learning
Amazon Kinesis
Generating Logs to stream them to Kinesis
Data WarehouseSemi-structured NoSQL Predictive Models Other AppsStreaming
Amazon Simple
Storage Service
Data Lake Archive
Log Generator
9. Amazon RedshiftAmazon Elastic MapReduce
Amazon Glacier
Amazon
DynamoDB
Amazon Machine
Learning
Amazon Kinesis
Combinational Services for Data analytics
Data WarehouseSemi-structured NoSQL Predictive Models Other AppsStreaming
Amazon Simple
Storage Service
Data Lake Archive
Log Generator
10. Launch Redshift
• Cluster Identifier: myname-game-dw
• Database Name: mynamegame
• Database Port: 5439 (default)
• Node Type: dc1.large
• Cluster Type: Single Node
• Number of Compute Nodes: 1 (required for multi-node)
17. Amazon RedshiftAmazon Elastic MapReduce
Amazon Glacier
Amazon
DynamoDB
Amazon Machine
Learning
Amazon Kinesis
What we’ve done so far.
Data WarehouseSemi-structured NoSQL Predictive Models Other AppsStreaming
Amazon Simple
Storage Service
Data Lake Archive
Log Generator
18. Copy Log data from S3 to Redshift
$ python inserter.py
20. Amazon RedshiftAmazon Elastic MapReduce
Amazon Glacier
Amazon
DynamoDB
Amazon Machine
Learning
Amazon Kinesis
Creating a new table in Redshift
Data WarehouseSemi-structured NoSQL Predictive Models Other AppsStreaming
Amazon Simple
Storage Service
Data Lake Archive
Log Generator
Creating summary tables from log table
22. Run Business Intelligence Tools
Amazon RedshiftAmazon Elastic
MapReduce
Amazon
Glacier
Amazon
DynamoDB
Amazon
Machine
Learning
Amazon Kinesis
Data WarehouseSemi-structured NoSQL Predictive Models Other AppsStreaming
Amazon
Simple
Storage
Service
Data Lake Archive
Log
Generator
24. Loading Streaming Data into Amazon Elasticsearch Service
Amazon RedshiftAmazon Elastic
MapReduce
Amazon
Glacier
Amazon
DynamoDB
Amazon
Machine
Learning
Amazon Kinesis
Data WarehouseSemi-structured NoSQL Predictive Models Other AppsStreaming
Amazon
Simple
Storage
Service
Data Lake Archive
Log
Generator
Creating summary tables from log table
Amazon Elasticsearch
Service
25. Launch Elasticsearch
• Go to AWS management console
• Launch Elasticsearch domain
• Set access policy to public open for Demo only
26. Loading Streaming Data into Amazon Elasticsearch Service
Amazon RedshiftAmazon Elastic
MapReduce
Amazon
Glacier
Amazon
DynamoDB
Amazon
Machine
Learning
Amazon Kinesis
Data WarehouseSemi-structured NoSQL Predictive Models Other AppsStreaming
Amazon
Simple
Storage
Service
Data Lake Archive
Log
Generator
Creating summary tables from log table
Amazon Elasticsearch
Service
AWS
Lambda
27. Creating and configuring Lambda function
• https://github.com/awslabs/amazon-elasticsearch-lambda-samples
• Download a sample JS file
• Install required Nodejs packages
• Modify ElasticSearch endpoint
• Zip all files including node_modules
• Upload zip file to Lambda function
• Set lambda role to access Elasticsearch
33. • A fully managed service for continuously querying streaming data using standard
SQL
• Use cases: Preprocessing streams / Most frequently occurring values Counting
distinct values / Simple alerts / Detecting anomalies on a stream / Post processing
in application stream
Real-time Log Analytics
Amazon Kinesis Analytics
34. Amazon Kinesis Analytics
Amazon RedshiftAmazon Elastic
MapReduce
Amazon
Glacier
Amazon
DynamoDB
Amazon
Machine
Learning
Amazon Kinesis
Data WarehouseSemi-structured NoSQL Predictive Models Other AppsStreaming
Amazon
Simple
Storage
Service
Data Lake Archive
Log
Generator
Creating summary tables from log table
Amazon Elasticsearc
Service
AWS
Lambda
Amazon
Kinesis
Analytics