Learning Objectives: - Understand how to build a real-time application monitoring system using network traffic logs - Learn how to enrich and aggregate network flow logs data using Amazon Kinesis - Learn how to visualize and analyze your network data to gain actionable insights

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Allan MacInnis, Solutions Architect
11/7/2017
Real-time Application Monitoring
with Amazon Kinesis and
Amazon CloudWatch

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Streaming data overview
• Services overview
• Application monitoring scenarios
• Application monitoring example

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Streaming data overview

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Most data is produced continuously
Mobile Apps Web Clickstream Application Logs
Metering Records IoT Sensors Smart Buildings

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The diminishing value of data
Recent data is highly valuable
• If you act on it in time
• Perishable Insights (M. Gualtieri,
Forrester)
Old + Recent data is more
valuable
• If you have the means to combine
them

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Processing real-time, streaming data
• Durable
• Continuous
• Fast
• Correct
• Reactive
• Reliable
What are the key requirements?
Ingest Transform Analyze React Persist

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Services overview

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis makes it easy to work with real-
time streaming data
Amazon Kinesis
Streams
• For Technical Developers
• Collect and stream data
for ordered, replayable,
real-time processing
Amazon Kinesis
Firehose
• For all developers, data
scientists
• Easily load massive
volumes of streaming data
into Amazon S3, Redshift,
ElasticSearch
Amazon Kinesis
Analytics
• For all developers, data
scientists
• Easily analyze data
streams using standard
SQL queries

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Streams
• Reliably ingest and durably store streaming data at low
cost
• Build custom real-time applications to process streaming
data

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Firehose
• Reliably ingest and deliver batched, compressed, and
encrypted data to S3, Redshift, and Elasticsearch
• Point and click setup with zero administration and
seamless elasticity

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Analytics
• Interact with streaming data in real-time using SQL
• Build fully managed and elastic stream processing
applications that process data for real-time visualizations
and alarms

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudWatch
Monitor your AWS resources in near real-time
Monitor custom, application-specific metrics
Monitor and store logs
Set alarms
View graphs and statistics
Monitor and react to resource changes

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudWatch Logs
Monitor and store logs
• Monitor logs from Amazon EC2 instances with
CloudWatch Logs Agent
• Archive logged data
• Use other AWS services as data source:
• Amazon Route 53 DNS queries
• AWS CloudTrail logged events
• VPC flow logs

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudWatch Logs Subscriptions
Deliver near real-time feed of log events to Kinesis or AWS
Lambda
log data
Application
CloudWatch
Logs
Kinesis
Lambda

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of Kinesis - CloudWatch Logs
subscription• Use Kinesis Firehose to persist log data to another
durable storage location: S3, Redshift, Elasticsearch
Service
• Use Kinesis Analytics to perform near real-time
streaming analytics on your log data:
• Anomaly detection
• Aggregation
• Use Kinesis Streams with a custom stream processing
application to apply business logic to your log data:
• Alternate data destinations
• Data enrichment

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application monitoring scenarios

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring application-specific metrics
• Use CloudWatch Agent to send application logs to
CloudWatch Logs
• Analyze stream with Kinesis Analytics application
• Persist raw log data to durable storage with Kinesis
Firehose
log data
CloudWatch
Logs
Kinesis
Streams
Kinesis
Analytics
Kinesis
Firehose
DynamoDB
• Active users over past 15 minutes?
• Top 10 articles read in the past 30 minutes?
• Filter unwanted log entries
S3
log data
Application

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring application-specific metrics
• Don’t plan to use CloudWatch Events or Alarms?
• Consider logging directly to Kinesis with the Kinesis Agent or
Kinesis APIs
log data
Kinesis
Streams
Kinesis
Analytics
Kinesis
Firehose
DynamoDB
• Active users over past 15 minutes?
• Top 10 articles read in the past 30 minutes?
• Filter unwanted log entries
S3
Application

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring access to AWS resources
• Deliver AWS CloudTrail events to CloudWatch logs
• Create Kinesis Streams subscription
• Analyze Stream with Kinesis Analytics application
event data event data
CloudWatch
Logs
Kinesis
Streams
CloudTrail Kinesis
Analytics
Kinesis
Firehose
DynamoDB
SNS
Top 20 API calls over 1 min window?
What service is getting called the most?
What IAM user is making the most calls?
S3

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring network activity
• Use VPC Flow Logs to get visibility into application
communication
• VPC Flow Log records contain network data that can be
analyzed
2 123456789010 eni-abc123de 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK
Source IP Address Destination IP Address Action
ACCEPT | REJECT

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring network activity
• Enrich source and destination data in near real-time
• Aggregate data by specific dimensions and persist
aggregated values
network logs network logs
CloudWatch
Logs
Kinesis
FirehoseVPC Flow Logs
DynamoDB
Kinesis
Analytics
S3
map IP addresses to
application names
DynamoDB

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitor network activity
Is something wrong with the network?
Service A Service A
Service
B
Service
B
Service
C
Account 1234567890, Zone us-east-1e
Service
D
Service
D
Service
D
Service
E
Service F
Account 0987654321, Zone eu-west-1a

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitor network activity
Is something wrong with the network?
Service A Service A
Service
B
Service
B
Service
C
Account 1234567890, Zone us-east-1e
Service
D
Service
D
Service
D
Service
E
Service F
Account 0987654321, Zone eu-west-1a
Bad
deployment?

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitor network activity
Is something wrong with the network?
Service A Service A
Service
B
Service
B
Service
C
Account 1234567890, Zone us-east-1e
Service
D
Service
D
Service
D
Service
E
Service F
Account 0987654321, Zone eu-west-1a
Network
problems?

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitor network activity
What are all the dependencies of an application?
MySQL Service A Redis
Expected outbound dependencies for Service A

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitor network activity
What are all the dependencies of an application?
MySQL Service A Redis
Identified outbound dependencies for Service A using traffic
logs
S3
Service
B
DynamoD
B

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Additional Resources
Learn more about Amazon Kinesis in our documentation.
To easily send data to Amazon Kinesis, use the Kinesis Data
Generator. For additional information, see Test Your Streaming Data
Solution with the New Amazon Kinesis Data Generator.
Learn more about Amazon CloudWatch in our documentation.
For more ideas about log monitoring, see Implement Serverless Log
Analytics Using Amazon Kinesis Analytics and Real-time Clickstream
Anomaly Detection with Amazon Kinesis Analytics.

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!