Learning Objectives: - Get an overview of streaming data and it's application in analytics and big data. - Understand the factors driving the accelerating transformation of batch processing to real-time. - Learn how you should plan for incorporating data streaming in your analytics and processing workloads. Business can now easily perform real-time analytics on data that has been traditionally analyzed using batch processing in data warehouses or using Hadoop frameworks, and react to new information in minutes or seconds instead of hours or days. In this webinar, Forrester analyst Mike Gualtieri and Amazon Kinesis GM Roger Barga will discuss this prevalent trend, it's business significance, and how you should plan for it. You will also learn about the AWS services that can help you get started quickly with real-time, streaming applications fore your analytics and big data workloads.

1. Streaming Analytics: The Future Of
Every Application
Mike Gualtieri, Principal Analyst
July 27, 2017 AWS Webinar
Twitter: @mgualtieri

2. #Data

3. Data is like a drop of rain.

4. It forms instantaneously in a cloud…

5. ...and travels far before it makes a ripple.

6. #Real-time

7. All data originates in real-time!

8. #

9. But, analytics to gain insights is
usually done much, much later.

10. 10© 2016 Forrester Research, Inc. Reproduction Prohibited
“As you look to improve your data processing and analytics capabilities, what aspect of
the implementation is most important to your business? Please select one.”
11%
11%
12%
16%
24%
25%
Quick turnaround on customer requests
More data availability
Expanded access to more business users (i.e., self-
service)
Low cost
Advanced analytics capabilities (e.g. predictive.
prescriptive, streaming)
Faster performance (time to value)
Faster time to value and advanced analytics
is most important to business
Base: 100 data science and data analytics leaders at enterprises within the US
Source: A commissioned study conducted by Forrester Consulting, April 2016

11. #WhyWait

12. Insights are perishable.

13. Real-time
insights
Operational
insights
Performance
insights
Strategic
insights
Insight: Shopping for
furniture
Action: Recommend
cleaning supplies
Insight: Profit lower than
goal
Action: Optimize price
Insight: Demand forecast
strong
Action: Increase inventory
Insight: Furniture demand
high
Action: Expand product line
TimetoAct
Perishability
Sub-second to
seconds
Seconds to
hours
Days to
weeks
Weeks to
years
Sub-second to
seconds
Seconds to
hours
Hours to
weeks
Weeks to
years
Enterprises must act on a range of perishable
insights to get value from data and analytics

14. Batch analytics operations take too long
BusinessValue
Time To Action
Data
originated
Analytics
performed
Insights
gleaned
Action
taken
Outdated
insights
Impotent or
harmful
actions
PositiveNegative
Decision
made
Poor
decision

15. Hyper-compress the analytics lifecycle to
maximize the value of data
BusinessValue
Time To Action
PositiveNegative
Maximum
Business
Value

16. © 2013 Forrester Research, Inc. Reproduction Prohibited 16
Real-time means highly perishable

17. How can you know if you should you make an
offer or send a gentle nudge right now?

18. How can you warn other drivers that the
road is slippery to avoid a crash right now?

19. What are movers and shakers saying about
equities that we cover right now?

20. How you detect customer SLA problems right now?

21. Streaming technology is necessary to detect
and act on real-time perishable insights.

22. #Streaming

23. ANALYTICS
STREAMINGTechnology that ingests, analyzes, and acts on
high throughput of data from live data sources
to identify patterns, detect urgent situations,
and automate immediate actions in real-time.

24. © 2013 Forrester Research, Inc. Reproduction Prohibited 24
Thinking in streams is different…
› Ingest
› Filter
› Transform
› Normalize
› Link
› Enrich
› Correlate
› Location/motion (geofencing)
› Time windows
› Temporal pattern detection (CEP)
› Business logic/rules execution
› Action interfaces
Continuous Ingestion Continuous Analytics

25. #Solutions

26. 110010011011001
0100100110
0100110011011
0100
Customerdata
Transactions
Webcontent
Documents
Understand any data that enterprises can
throw at it.

27. Scale to handle spikes.

28. Protects confidential information

29. Provide fault-tolerance for mission-critical
business and customer applications.

30. Offer sophisticated state-full and stateless
analytics.

31. Leverage existing skills to make it easy for
developers to develop, test, and deploy
applications.

32. #Discussion

33. Why is streaming in
the cloud gaining in
popularity?

34. Why AWS customers use Amazon Kinesis?
Lower costs
Performant without
heavy lifting
Scales elastically
Increased agility
Secure and visible
Plug and play

35. What does it take to
implement
streaming analytics?

37. Amazon Kinesis
Kinesis FirehoseKinesis AnalyticsKinesis Streams
DeliverProcessCapture

38. Amazon Kinesis
Kinesis FirehoseKinesis AnalyticsKinesis Streams
DeliverProcess
For Technical Developers
Build your own custom
applications that process
or analyze streaming data

39. Amazon Kinesis
Kinesis FirehoseKinesis AnalyticsKinesis Streams
Deliver
For Technical Developers
Build your own custom
applications that process
or analyze streaming data
For All Developers, Data
Scientists, Business Analysts
Easily analyze data streams
using standard SQL queries

40. Amazon Kinesis
Kinesis FirehoseKinesis AnalyticsKinesis Streams
For Technical Developers
Build your own custom
applications that process
or analyze streaming data
For All Developers, Data
Scientists, Business Analysts
Analyze data streams using
standard SQL queries
For All Developers
Load streaming data into S3,
Redshift and Elasticsearch,
clean and format using Lambda

41. How does streaming
analytics work?

42. Load Streaming Data into Data Stores

43. Process Streaming Data in Real-time with SQL

44. Build Custom Applications using Popular
Stream Processing Frameworks

45. • Write SQL code to process streaming data
• Connect to streaming source
• Continuously deliver SQL results
Kinesis Analytics – How it Works

46. Connect to Streaming Source
• Streaming data sources include Kinesis Firehose or
Kinesis Streams
• Input formats include JSON, .csv, variable column,
unstructured text
• Each input has a schema; schema is inferred, but you
can edit
• Reference data sources (S3) for data enrichment

47. Write SQL Code
• Build streaming applications with one-to-many SQL
statements
• Robust SQL support and advanced analytic functions
• Extensions to the SQL standard to work seamlessly
with streaming data
• Support for at-least-once processing semantics

48. Continuously deliver SQL results
• Send processed data to multiple destinations
• S3, Amazon Redshift, Amazon ES (through Firehose)
• Streams (with AWS Lambda integration for custom
destinations)
• End-to-end processing speed as low as sub-second
• Separation of processing and data delivery

49. Simple counting (e.g. failure count)
Counting with Windows ( e.g. failure count every hour)
Preprocessing: filtering, transformations (e.g. data cleanup)
Alerts, thresholds (e.g. Alarm on high temperature)
Detect Event Sequence Patterns – small transaction followed by large
transaction, etc.
Tracking - follow some related entity’s state in space, time etc. (e.g. location
of airline baggage, vehicle, tracking wild life)
Detect trends – Rise, turn, fall, outliers, complex trends like triple bottom
etc., (e.g. algorithmic trading, SLA, load balancing)
Real-time Analytics over Streaming Data

51. • Compute key performance indicates over time windows
• Combine with historical data in Amazon S3 or Redshift
Kinesis
Analytics
Kinesis
Streams
Kinesis
Firehose
Amazon
Redshift
Amazon
S3
Kinesis
Streams
Kinesis
Firehose
Custom ,
Real-time
Destinations
Generate Time Series Analytics

52. • Validate and transform raw data, process to calculate meaningful statistics
• Send processed data to visualize in BI and visualization services
Amazon
QuickSight
Kinesis
Analytics
Amazon
Elasticsearch
Service
Amazon
Redshift
Amazon
RDS
Kinesis
Streams
Kinesis
Firehose
Feed Real-time Dashboards

53. • Build sequences of events from stream, such as user sessions in a
clickstream or app behavior through logs
• Identify events of interest and take action through alarms & notifications
Kinesis
Analytics
Kinesis
Streams
Kinesis
Firehose
Kinesis
Streams
Amazon
SNS
Amazon
CloudWatch
AWS
Lambda
Create Real-time Alarms & Notifications

54. What market
segments are using
streaming?

55. Accelerated Ingest-
Transform-Load
Continuous Metrics
Generation
Machine Learning and
Actionable Insights
Ad Tech/
Marketing
Publisher, bidder data
aggregation
Advertising metrics like
coverage, yield, and
conversion
User engagement with
ads, optimized bid/buy
engines
IoT Sensor, device telemetry
data ingestion
Operational metrics and
dashboards
Device operational
intelligence and alerts
Gaming Online data aggregation,
e.g., top 10 players
Massively multiplayer
online game (MMOG) live
dashboard
Leader board generation,
player-skill match
Consumer
Online
Clickstream analytics Metrics like impressions
and page views
Recommendation engines,
proactive care
Operations
Monitoring
DevOps tools, ingesting
VPCFlowLogs
Subscribe to CloudWatch
Logs and analyze logs in
real-Time
Anomaly detection
Streaming Data Scenarios Across Verticals

56. What are some
specific examples?

57. Kaiten Sushiro’s wonderful world of streaming data

58. • Kaiten Sushi Chain restaurant
• Stream sensor data from Kaiten Sushi to Kinesis to improve operations.
Kaiten Sushiro’s wonderful world of streaming data

59. Amazon Game Studios’
wonderful world of streaming data…

60. Amazon Game Studios’
Analyze massive amounts of gameplay data from players
Stream data for entire player base
Process in real-time
Visualize (SQL queries)
Archive player data for years

61. Hearst’s wonderful world of streaming data

62. Buzzing@Hearst

63. The Business Value of Buzzing@Hearst
Real-Time Reactions
Instant feedback on articles from their audiences
Promoting Popular Content Cross-Channel
Incremental re-syndication of popular articles across properties
(e.g. trending newspaper articles can be adopted by magazines)
Authentic Influence
Inform Hearst editors to write articles that are more
relevant to their audiences
Understanding Engagement
Inform both editors what channels their audiences are
leveraging to read Hearst articles
INCREMENTAL
REVENUE
25% more
page views
15% more
visitors

64. 1 billion events/wk from
connected devices
17 PB of game data per
season
Real-time home estimates
on 100M+ homes
100 GB/day click streams
from 250+ sites
50 billion ad
impressions/day
sub-50 ms responses
10 million events/day
Ingesting 2M+ network
events every second
Funnel all
production events
through Kinesis
Amazon Kinesis Customer Examples

65. #Time

66. Stop wasting it.

67. Use it to your advantage.

68. How to Get Started
• Read our streaming analytics whitepaper:
https://aws.amazon.com/whitepapers/kinesis-solutions/
• Visit the Kinesis website:
https://aws.amazon.com/kinesis/
• Try this hands-on tutorial:
https://aws.amazon.com/getting-started/projects/build-
log-analytics-solution/

69. forrester.com
Thank you
Mike Gualtieri
mgualtieri@forrester.com
Twitter: @mgualtieri