Organisations today need a way to manage the ever-increasing volume of data from numerous sources such as log systems, click streams or connected devices and be able to analyse this data in real-time. In this session we will walk through an architecture demonstration of how to leverage AWS services to meet these needs. Speaker: Ganesh Raja, Solutions Architect, Amazon Web Services

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Ganesh Raja
Solutions Architect, Amazon Web Services
Building a Real-­Time Dashboard
Amazon Kinesis, AWS Lambda and Amazon Elasticsearch Service
Technical 201

2. Agenda
• Streaming Big Data
• Introducing Kinesis, Lambda and Elasticsearch
• Building the Real-­Time Dashboard

3. You’re Likely Already “Streaming”
Sensor networks analytics
Ad network analytics
Log shipping and centralisation
Click stream analysis
Gaming status
Hardware and software appliance metrics
…more…

4. Let’s Explore Common Challenges
of Streaming

5. One common starting point is ingesting records
for analysis
Elastic Beanstalk
foo-­analysis.com
Global top-­10
foo-­analysis.com

6. Too big to handle on one box
Global top-­10Elastic Beanstalk
foo-­analysis.com

7. Too big to handle on one box
Global top-­10Elastic Beanstalk
foo-­analysis.com

8. When to use Stream Processing
• “real-­time” starts coming onto the radar
• The time to answer can’t wait for batch processing times
• Instead of processing serially as A > B > C it would be
better to have a fan out pattern
• The records are just a means to an end, most records
can be immediately archived after an “answer” is
determined.

9. Kinesis

10. • For Technical Developers
• Build your own custom applications
that process or analyze streaming
data
• GA at re:Invent 2013
Amazon Kinesis: Streaming Data Made Easy
Amazon Kinesis Streams

11. Amazon Kinesis
Data
Sources
Data
Sources
Data
Sources
Data
Sources
Data
Sources
AWS Endpoint
Availability
Zone
Shard 1
Shard 2
Shard N
Availability
Zone
Availability
Zone
App.4
[Machine
Learning]
App.1
[Aggregate
& De-­
Duplicate]
App.2
[Metric
Extraction]
App.3
[Sliding
Window
Analysis]
S3
DynamoDB
Redshift
Elastic Search

12. Sending & Reading from Kinesis Streams
AWS SDK
LOG4J
Flume
Fluentd
Get* APIs
Kinesis Client Library
+
Connector Library
Apache Storm
Amazon Elastic
MapReduce
Sending Consuming
AWS Mobile
SDK
Kinesis
Producer
Library
AWS Lambda
Apache Spark

13. • For Technical Developers
• Build your own custom applications
that process or analyze streaming
data
• GA at re:Invent 2013
Amazon Kinesis: Streaming Data Made Easy
Services make it easy to capture, deliver and process streams on AWS
Amazon Kinesis Streams

14. • For all developers, data scientists
• Easily load massive volumes of
streaming data into S3 and Redshift
• GA at re:Invent 2015
Amazon Kinesis: Streaming Data Made Easy
Amazon Kinesis Firehose

15. Amazon Kinesis Firehose
Load massive volumes of streaming data into Amazon S3 and Amazon Redshift
Zero administration
Direct-­to-­data store integration
Seamless elasticity
Capture and submit
streaming data to Firehose
Analyze streaming data using your
favorite BI tools
Firehose loads streaming data
continuously into S3 and Redshift

16. • For all developers, data scientists
• Easily analyze data streams using
standard SQL queries
• Coming soon
Amazon Kinesis: Streaming Data Made Easy
Amazon Kinesis Analytics

17. • Custom Processing
• Sub Second Response Time
• Complete control over Streams and Processing workflows
• Zero Administration
• Data Latency of over 60 Seconds
• Durable Delivery of data from Source to S3 / Redshift

18. What is Sonos?
Sonos is the smart
speaker system that
streams all your
favorite music to any
room, or every room.

19. Control your music with
one simple app, and fill
your home with pure,
immersive sound.
What is Sonos?

20. A Plethora of Data Sources

21. Firmware device logs
Application telemetry
Music service usage metrics
Cloud applications logs
Performance Indicators
Where does all this Wonderful Data Come From?

22. Manufacturing tests and yields
Diagnostics
Customer support
Sales and marketing data
Diagnostics
Where does all this Wonderful Data Come From?

23. Sonos’ Streaming Data Architecture
Collect Store Process Consume
100K+ device
Install base
1.5 B Events
3 TB/ day
25 Shards
2KB / event
[5] C4.2XL
KCL

24. Revisit our Streaming Application
Global top-­10Elastic Beanstalk
foo-­analysis.com
Data
Record
Stream
Shard
Partition Key
Worker
My top-­10
Data Record
Shard:
Sequence Number
14 17 18 21 23

25. Lambda

26. AWS Lambda
Serverless, event-­driven compute service
Lambda = microservice without servers

27. AWS Lambda – Benefits
EVENT-­DRIVEN SCALESERVERLESS SUBSECOND BILLING

28. AWS Lambda – Capabilities
SIMPLE RESOURCE MODEL
FLEXIBLE INVOCATION PATHS GRANULAR PERMISSIONS CONTROL
BRING YOUR OWN CODE

29. AWS Lambda – Use Cases
Control Systems
Customize responses
and response workflows
to state and data
changes within AWS
Data Processing
Execute code in
response to changes in
data, shifts in system
state, or actions by users
Backends
Execute backend logic to
handle requests for web,
mobile, IoT, and 3rd APIs

30. More Triggers for AWS Lambda
AWS
CloudFormation
Custom Resources
Amazon SES
Actions
Amazon Cognito
Sync Triggers
… and the list will
continue to grow!
Amazon SWF
Tasks
Amazon Echo
Skills
AWS Lambda
Scheduled Events
AWS IoT Actions

31. Common Integration Pattern with Kinesis

32. Easy Real-­Time Streaming Architecture
Data Ingestion
with Amazon
Kinesis
(PUT record)
Records
retrieved by
AWS Lambda
Your code runs
once per [batch of]
records
Amazon S3
Amazon
DynamoDB
Smart
Devices
Click
Stream
Log
Data
Amazon
Redshift

33. Revisit our Streaming Application
Global top-­10Elastic Beanstalk
foo-­analysis.com
Data
Record
Stream
Shard
Partition Key
Worker
My top-­10
Data Record
Shard:
Sequence Number
14 17 18 21 23

34. Elasticsearch as a Service

35. A powerful, real-­time, distributed, open-­source search and
analytics engine:
• Built on top of Apache Lucene
• Schema free
• Developer friendly RESTful API

36. Combined with Logstash and Kibana, the ELK stack
provides a tool for real-­time analytics and data visualisation

37. Introducing Amazon Elasticsearch Service
• Managed Service
• Easily Operate and Scale

38. Key benefits
Easy cluster
creation and
configuration
management
Support for ELK Security with AWS
IAM
Monitoring with
Amazon
CloudWatch
Auditing with AWS
CloudTrail
Integration options
with other AWS
services
(CloudWatch Logs,
Amazon
DynamoDB,
Amazon S3,
Amazon Kinesis)

39. Revisit our Streaming Application
Global top-­10
Data
Record
Stream
Shard
Partition Key
Worker
My top-­10
Data Record
Shard:
Sequence Number
14 17 18 21 23

40. Building the Real-­time Visualisation.

41. Building the Real-­time Visualisation

42. Demo Time

43. Building the Real-­time Visualization

44. AWS Training & Certification
Intro Videos & Labs
Free videos and labs to
help you learn to work
with 30+ AWS services
– in minutes!
Training Classes
In-­person and online
courses to build
technical skills –
taught by accredited
AWS instructors
Online Labs
Practice working with
AWS services in live
environment –
Learn how related
services work
together
AWS Certification
Validate technical
skills and expertise –
identify qualified IT
talent or show you
are AWS cloud ready
Learn more: aws.amazon.com/training

45. Your Training Next Steps:
ü Visit the AWS Training & Certification pod to discuss your
training plan & AWS Summit training offer
ü Register & attend AWS instructor led training
ü Get Certified
AWS Certified? Visit the AWS Summit Certification Lounge to pick up your swag
Learn more: aws.amazon.com/training

46. Thank you!