Learning Objectives: - Understand how to easily build an end to end, real time log analytics solution - Get an overview of collecting and processing data in real-time using Amazon Kinesis - Learn how to Interactively query and visualize your log data using Amazon Elasticsearch Service Log analytics is a common big data use case that allows you to analyze log data from websites, mobile devices, servers, sensors, and more for a wide variety of applications such as digital marketing, application monitoring, fraud detection, ad tech, gaming, and IoT. Moving your log analytics to real time can speed up your time to information allowing you to get insights in seconds or minutes instead of hours or days. In this session, you will learn how to ingest and deliver logs with no infrastructure using Amazon Kinesis Firehose. We will show how Amazon Kinesis Analytics can be used to process log data in real time to build responsive analytics. Finally, we will show how to use Amazon Elasticsearch Service to interactively query and visualize your log data.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Ray Zhu, Sr. Product Manager, Amazon Kinesis
6/13/2017
Real-Time Log Analytics using
Amazon Kinesis and Amazon
Elasticsearch Service

2. Agenda
• Amazon Kinesis
• Amazon Elasticsearch Service
• Real-time Log Analytics
o Step 1 Create Amazon Elasticsearch Cluster
o Step 2 Set Up Firehose Delivery Stream
o Step 3 Send Data to Firehose Delivery Stream
o Step 4 Analyze the Data from Amazon Elasticsearch Service
o Step 5 Monitor Log Ingestion Pipeline

3. Amazon Kinesis
Amazon Kinesis makes it easy to collect, process,
and analyze real-time, streaming data so you can
get timely insights and react quickly to new
information.

4. Kinesis Streams
Stores data as a
continuous replayable
stream for custom
applications
Kinesis Firehose
Load streaming data into
Amazon S3, Amazon
Redshift, and Amazon
Elasticsearch Service
Kinesis Analytics
Analyze data streams
using standard SQL
queries
Amazon Kinesis

5. Kinesis Firehose Benefits

6. Amazon Elasticsearch Service
A cost-effective managed service that makes it easy to
deploy, manage, and scale open source Elasticsearch for
log analytics, full-text search and more.

7. Amazon Elasticsearch Service Benifits

8. Time Value of Money Data

9. Real-Time Log Pipeline
Elasticsearch
cluster
backup S3 bucket
source records
data source
source records
Firehose
delivery stream
transformed
records
delivery failure
transformation failure

10. Step 1 Create Amazon
Elasticsearch Cluster

11. Define Domain

12. Configure Cluster

13. Access Policy

14. Step 2 Set Up Firehose
Delivery Stream

15. Destination

16. Configuration

17. Configuration

18. Step 3 Send Data to Firehose
Delivery Stream

19. Sample Log Data
219.134.32.117 - - [16/Feb/2017:09:38:20 -0800] "GET /wp-content HTTP/1.1" 200 4521
"-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/5.1; .NET CLR
3.8.23015.5)“
95.169.41.62 - - [16/Feb/2017:09:38:20 -0800] "PUT /app/main/posts HTTP/1.1" 200
3883 "-" "Mozilla/5.0 (Windows NT 6.2; Trident/7.0; rv:11.0) like Gecko“
221.147.191.247 - - [16/Feb/2017:09:38:20 -0800] "GET /explore HTTP/1.1" 200 6579 "-"
"Mozilla/5.0 (Windows; U; Windows NT 5.1) AppleWebKit/538.0.1 (KHTML, like Gecko)
Chrome/38.0.895.0 Safari/538.0.1“
179.96.123.130 - - [16/Feb/2017:09:38:20 -0800] "GET /list HTTP/1.1" 200 560 "-"
"Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:5.4) Gecko/20100101 Firefox/5.4.6“
132.119.12.76 - - [16/Feb/2017:09:38:20 -0800] "PUT /explore HTTP/1.1" 200 3131 "-"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_0 rv:5.0; AZ) AppleWebKit/535.1.0
(KHTML, like Gecko) Version/4.0.3 Safari/535.1.0"

20. After Data Transformation
{"host":"210.197.211.230","ident":"-","authuser":"-","request":"GET /explore
HTTP/1.1","response":301,"bytes":2457,"verb":"GET","@timestamp":"2017-06-
07T14:35:32.000Z","timezone":"-0700","@timestamp_utc":"2017-06-07T21:35:32.000Z"}
{"host":"25.217.150.148","ident":"-","authuser":"-","request":"PUT /app/main/posts
HTTP/1.1","response":200,"bytes":662,"verb":"PUT","@timestamp":"2017-06-
07T14:35:32.000Z","timezone":"-0700","@timestamp_utc":"2017-06-07T21:35:32.000Z"}
{"host":"187.232.212.221","ident":"-","authuser":"-","request":"GET /app/main/posts
HTTP/1.1","response":200,"bytes":1549,"verb":"GET","@timestamp":"2017-06-
07T14:35:32.000Z","timezone":"-0700","@timestamp_utc":"2017-06-07T21:35:32.000Z"}
{"host":"227.167.15.81","ident":"-","authuser":"-","request":"PUT /explore
HTTP/1.1","response":200,"bytes":7591,"verb":"PUT","@timestamp":"2017-06-
07T14:35:32.000Z","timezone":"-0700","@timestamp_utc":"2017-06-07T21:35:32.000Z"}

21. Send Log Data
https://awslabs.github.io/amazon-kinesis-data-generator/web/producer.html

22. Step 4 Analyze the Data from
Amazon Elasticsearch Service

23. Query Data from Kibana

24. Visualize Data from Kibana

25. Step 5 Monitor Log Ingestion
Pipeline

26. Monitor with CloudWatch Metrics

27. Monitor with CloudWatch Logs

28. Q & A

29. Thank you!