Explore the various options for streaming data on AWS, such as Amazon Kinesis and Amazon Managed Streaming for Kafka, and the various options for processing streams of data such as Apache Spark, Apache Flink, AWS Lambda, and Amazon Kinesis Analytics for Java. Let's explore what an architecture for processing Australia's new Open Banking data format at 60,000 transactions per second could look like.

1. S U M M I T
SYDNEY

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Stream processing in 2019
Chris Horder
Solutions Architect
Amazon Web Services

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Typical credit card transaction
Credit Card
Institution
User
Merchant Mobile
client

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Mobile Apps Web Clickstream Application Logs
Metering Records IoT Sensors Smart Cities
[Wed Oct 11 14:32:52
2018] [error] [client
127.0.0.1] client
denied by server
configuration:
/export/home/live/ap/ht
docs/test
Data is produced continuously

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The diminishing value of data over time
Historical
Reactive
Actionable
Preventive/
Predictive
Informationin
Decision-Making
Time-critical
Decisions
Traditional“Batch”
BusinessIntelligence
Months…DaysHoursMinutesSecondsRealTime
ValueofDatatoDecision-Making

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Customer examples
50 billion daily ad
impressions, sub-
50-ms responses
1 billion events per
week from connected
devices
Facilitate
communications
between 100+
microservices
Analyse billions of
network flows in
real time
Near-real-time home
valuation
(Zestimates)
Live clickstream
dashboards refreshed
under 10 sec.
IoT predictive
analytics
Analyse game
events in near real
time

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What do we need from our streaming architecture
Event driven
Source for batch
Consumption models
Schema flexibility
Microservices
Loosely coupled
Elasticity
Horizontal
scalability
Operations

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Components of a streaming architecture
Producer Message Buffer
Topic A
Topic B
Consumer
Producer
Producer
Producer
Producer
Producer
Schema
Repository
Consumer
Consumer
Consumer
Consumer

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Checkpoint: Amazon SQS, Amazon Kinesis and
Amazon Managed Streaming for Kafka
• Traditional Messaging
semantics
• Transparent scaling
• Individual message
delay
• Dead letter queues
• Multiple Consumers
• Native AWS
Integrations
• Fully Managed
• Control over ordering
• Highly configurable
retention
• Managed Kafka and
Zookeeper
• Existing applications
• Full configurability
• Log compaction
Amazon Simple
Queue Service
Amazon
Kinesis
Amazon Managed
Streaming for Kafka
Kafka on Elastic Compute
Cloud (EC2)

10. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Producers: AWS native
AWS Database Migration
Service
Kinesis Producer Library
Kinesis Agent
AWS IoT Amazon Connect
Amazon Pinpoint
Amazon DynamoDB
Streams
AWS Tools and SDKs
Amazon API Gateway
Amazon EMR

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Producers: Partners and SDKs

13. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Real-time
Fully managed
Scalable
Secure
Cost effective
Amazon EMR/Spark
Custom code on
Amazon EC2
Amazon S3
Amazon
Redshift
Splunk
Ingest,
store data
streams
Amazon
Kinesis Data
Streams
Amazon
Kinesis Data
Analytics
Aggregate,
filter, enrich
data
Amazon
Kinesis Data
Firehose
Egress
data
streams
AWS Lambda
Amazon
Elasticsearch
Service
Amazon Kinesis Data Streaming
Collect Process and analyse data streams in real-time

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Our Broker: Apache Kafka

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Apache Kafka anatomy 101
Producer
Broker
Broker
Broker
Consumer
Cluster
Zookeeper
Producer

17. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Consumers
Kinesis Consumer Library
Kinesis Agent
AWS Lambda
Amazon EMR Amazon Kinesis
Data Analytics
Amazon Kinesis
Data Firehose
SDK’s

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Apache Flink
Framework and distributed engine for stateful processing of data streams.
Simple programming High performance Stateful Processing Strong data integrity
Easy to use and flexible
APIs make building
apps fast
In-memory computing
provides low latency &
high throughput
Durable application
state saves
Exactly-once processing
and consistent state

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Use cases
Using data from multiple providers
Bank
Account
Institution
Credit Card
Institution
User
Value add
Insights across multiple
providers
Institution A - Feed
Institution B - Feed
Consumer Trend
Analysis
Product
Recommendation
Fraud Analysis
Merchant
Merchant - Feed
Mobile
client
Users

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Getting started
Amazon Managed
Streaming for Kafka
Amazon Elasticsearch
Service
{
"accountId": "11185",
"transactionId": 1155776,
"isDetailAvailable": "true",
"type": ”PAYMENT",
"status": ”POSTED",
"description": "Credit card payment to Sample Shop",
"postingDateTime": 1553207924271,
"valueDateTime": "22-03-2019 09:38:44",
"executionDateTime": 1553207924271,
"amount": 767.8318,
"currency": "AUD",
"reference": "AKG62DHBB5",
"merchantName": ”Sample shop : Purchase",
"merchantCategoryCode": "4112"
}`

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Getting started: Infrastructure
VPC
AWS Cloud
Availability Zone 1 Availability Zone 2
Instance
Broker
Instance
Broker
JMeter
Node Node
Availability Zone 3
Instance
Broker
Node
Amazon
ECS for
Kubernetes
Amazon
Managed
Streaming for
Kafka

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Sample data
{
"accountId": "11158",
"transactionId": “126649”,
"isDetailAvailable": true,
"type": "PAYMENT",
"status": "POSTED",
"description": "Credit card payment to AccountY",
"postingDateTime": "09-04-2019 17:17:44",
"valueDateTime": "09-04-2019 17:17:44",
"executionDateTime": "09-04-2019 17:17:44",
"amount": 534.24,
"currency": ”AUD",
"reference": "BF6GCJ2EDE",
"merchantName": ”MerchantA",
"merchantCategoryCode": "3150"
}

24. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Flink console

26. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SQL over the stream
Table result1 =
tableEnv.sqlQuery("select transactionId," +
" amount, " +
" accountId, " +
" currency, " +
" type " +
"from Transactions " +
" where currency = 'BAD' and amount > 995 and type = 'TRANSFER_OUTGOING'"
);
• Familiar syntax
• Powerful where clause
• Joins across streams
• Aggregations

27. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Aggregations and windowing
select TUMBLE_START(created_date, INTERVAL '20' SECOND) as wStart,
TUMBLE_END(created_date, INTERVAL '20' SECOND) as wEnd,
SUM(amount),
AVG(amount),
COUNT(*)
from Transactions
GROUP BY TUMBLE(created_date, INTERVAL '20' SECOND)
Tumbling
Sliding
Session

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Categorise transactions from a stream
Amazon Managed
Streaming for Kafka
Mobile
client
Users
Amazon Simple
Notification Service
Amazon API GatewayAmazon DynamoDB
Amazon SageMaker Amazon Managed
Streaming for Kafka

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Grow with your requirements
Amazon Managed
Streaming for Kafka
Mobile
client
Amazon API Gateway
Amazon Simple Storage
Service (S3)
Amazon Athena
Amazon RedshiftAmazon Elasticsearch
Service
DesktopAmazon RDS
AWS LambdaAmazon Neptune
Amazon EC2
Analyst

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Grow with your requirements
Amazon Managed
Streaming for Kafka
Mobile
client
Amazon API Gateway
Amazon Simple Storage
Service (S3)
Amazon Athena
Amazon RedshiftAmazon Elasticsearch
Service
DesktopAmazon RDS
AWS LambdaAmazon Neptune
Amazon EC2
Analyst

31. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Data61 - CSIRO
• Standards developed as part of the introduction in Australia of the Consumer
Data Right legislation to give Australians greater control over their data
• The Consumer Data Right is intended to apply sector by sector across the whole
economy, beginning in the banking, energy and telecommunications sectors.
• https://consumerdatastandardsaustralia.github.io

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Resources
• Free online training - https://www.aws.training
• Flink Homepage - https://flink.apache.org/
• AWS Big Data Blog - https://aws.amazon.com/blogs/big-data/
• Real-time bushfire alerting with Complex Event Processing in Apache Flink on Amazon EMR and IoT sensor
network
• Confluent - https://www.confluent.io/blog/
• Designing Data-Intensive Applications by Martin Kleppmann

34. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Chris Horder
chorder@amazon.com