Everyone has logs. They’re not the most exciting data that your systems generate, but often, they are the most useful. Across the board, we see customers using Amazon Elasticsearch Service (Amazon ES) to ingest, analyze, and search their log data. In this chalk talk, we discuss how to get your data into Amazon ES, and how to use Kibana to best effect to pull the information you need from the logs you’re generating.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What Can Your Logs Tell You?
Pranav Nambiar
Senior Manager (PM)
AWS
A N T 2 1 5

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why log analytics?

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The explosion of machine-generated data
10X
Faster Growth
Transition from IT to
DevOps
IoT and Mobile
Devices
Cloud-based
architectures

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sources of data
Gaming IOT sensorsDevices
External
systems
and
applications Web content
Logs, logs, and
more logs …
Databases Servers NetworkingStorage
Internal
systems
and
applications

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges
ahead
Searchability
Visualization
Timeliness
Integration
Cost
Unstructured
[Wed Oct 11 14:32:52
2000] [error] [client
127.0.0.1] client denied
by server configuration:
/export/home/live/ap/htd
ocs/test
Volume
There is more to log analytics than using a
database/data warehouse

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The problem with log files
Traditional approach = more time, less accurate, negative impact to business

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The solution

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Elasticsearch Service

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Elasticsearch Service is a
fully managed service that makes
it easy to deploy, manage, and
scale Elasticsearch and Kibana
Amazon Elasticsearch Service

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Benefits of Amazon Elasticsearch Service
Supports open-source
APIs and tools
Drop-in replacement with no need
to learn new APIs or skills
Easy to use
Deploy a production-ready
Elasticsearch cluster in minutes
Scalable
Resize your cluster with a few
clicks or a single API call
Secure
Deploy into your VPC and restrict
access using security groups and IAM
Highly available
Replicate across Availability Zones,
with monitoring and automated
self-healing
Integrated with
other AWS services
Seamless data ingestion, security,
auditing, and orchestration

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Leading Elasticsearch use-cases
Application monitoring & root-
cause analysis
Security information and event
management (SIEM)
IoT & mobile Business & clickstream analytics

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Discussion

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you set up your Amazon
Elasticsearch Service domain?

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS SDK
AWS Command Line
Interface (AWS CLI)
AWS CloudFormation
Elasticsearch
data nodes
Elasticsearch
master nodes
Elastic Load
Balancing
IAM
Amazon
CloudWatch
AWS
CloudTrail
Amazon Elasticsearch Service domain
Service architecture

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Create a domain

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Index size is approximately source
size
• Double this if you are deploying an
index replica
• Instance count based on storage
requirements
• Either local storage or up to 1.5 TB of
Amazon Elastic Block Store (Amazon
EBS) per instance
A 1 TB corpus will need two instances
Assuming a replica and using Amazon EBS
Given 1.5 TB of storage per instance, this gives 3 TB of storage
How many instances?

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Instance Max storage Workload
T2 3.5 TB You want to do dev/QA
M3, M4 150 TB Your data and queries are “average”
R3, R4 150 TB You have higher request volumes, larger
documents, or are using aggregations heavily
C4 150 TB You need to support high concurrency
I2, I3 1.5 PB You have XL storage needs with high IOPS
requirement
Which instance type?

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How to ingest your data?

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data source Collect TransformBuffer Deliver
Ingestion pipeline tasks

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Firehose
CloudWatch logs
Logstash
AWA Lambda
Transform
Amazon Kinesis Data Firehose/
Amazon Kinesis Data Streams
CloudWatch logs
Amazon
ElastiCache/Redis
Kafka Rabbit MQLogstash
Amazon Simple Storage
Service (Amazon S3)
Buffer
Data Firehose/
Kinesis Data Streams
Logstash
Worker nodes
Deliver
Amazon Kinesis agent
CloudWatch logs
agent
Beats
Fluentd
Application
Logstash
Collect
CloudWatch logs
AWS Lambda

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example solution
Applicationtraffic
Amazon Elasticsearch
Service
Filebeat
Filebeat tails access.log
and ships to Redis
Redis buffer
Logstash transforms
and delivers
On the application host
application.log

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Firehose solution
Applicationtraffic
application.log
Kinesis agent collects
log lines
On the application host
Kinesis Agent
Kinesis Data Firehose
Kinesis Data Firehose buffers,
transforms, and delivers
Amazon Elasticsearch
Service

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Files
S3 events
Amazon S3 AWS Lambda
function
Amazon
DynamoDB
streams
Amazon
DynamoDB
table
AWS Lambda
function
Amazon Elasticsearch Service
Amazon KinesisData
producers
AWS Lambda
function
Lambda architectures to ingest data

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How to analyze your data?

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Each log line or other event constitutes a search document
Each log line has various fields that contain data you can search and analyze

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Elasticsearch documents: Structured JSON
{ "verb": "GET",
"ident": "-",
"bytes": 6245,
"@timestamp": "1995-07-01T00:00:01",
"request":
"GET /history/apollo/ HTTP/1.0",
"host": "199.72.81.55",
"authuser": "-",
"@timestamp_utc":
"1995-07-01T04:00:01+00:00",
"timezone": "-0400",
"response": 200
}
• Documents contain fields – name/value
pairs
• Fields can nest
• Value types include text, numerics,
dates, and geo objects
• Field values can be single or array
• When you send documents to
Elasticsearch they should arrive as
JSON
199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Categorize your data using charts
xx.xxx.xxxx
xx.xxx.xxxx
xx.xxx.xxxx
Categorized data
My dashboard 1

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Set up dashboards

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
System 1
System 2 System 4
System 3
Response <> 200
Response <> 200 Response <> 200
Response <> 200
Timelion charts ‒ Analyze events across multiple systems

31. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.