This document provides an overview of AWS Kinesis and its components for streaming data. It discusses Kinesis Streams for ingesting and processing streaming data at scale. Kinesis Streams uses shards to provide throughput capacity. To ingest 10,000 records per second of 512 byte size each would require a Kinesis stream configured with 10 shards. Kinesis Firehose is for delivering streaming data to destinations like S3 or Redshift. Kinesis Analytics allows running SQL queries on streaming data and processing it in real-time.

1. AWS KINESIS
SCRATCHING THE SURFACE OF

2. QUESTION TIME

3. QUESTION
What (Kafka?) cluster do you need
to ingest:
- 10,000 record/sec
- record size 512Bytes

4. KINESIS

5. AWS KINESIS
WHAT IS KINESIS?
▸ Platform for streaming data on AWS
▸ "sometimes TBs per hour"

6. AWS KINESIS
MOTIVATION
▸ Highly Scalable
▸ Durable
▸ Elastic
▸ Replay-able reads

7. AWS KINESIS
KINESIS COMPONENTS
Firehose AnalyticsStreams

8. KINESIS STREAMS

9. AWS KINESIS STREAMS

10. AWS KINESIS STREAMS
TERMINOLOGY
▸ Streams - ordered sequence of data records
▸ Data record - Sequence Number, Partition Key, Data Blob
▸ 1MB max
▸ Retention period - 24h - 7d
▸ Producers, Consumers
▸ Shards

11. AWS KINESIS STREAMS
KINESIS STREAMS - SHARDS
▸ Fixed unit of capacity
▸ Read
▸ 5 transaction / sec
▸ 2MB / sec
▸ Write
▸ 1000 records / sec
▸ 1MB / sec

12. QUESTION
What cluster do you need to ingest
- 10,000 records/sec
- record size 512Bytes

13. AWS KINESIS STREAMS
KINESIS STREAMS - SHARD CALCULATION
Requirement
Kinesis Stream Write
Capacity
10,000 records/sec 1,000 Record/sec
512 Bytes/rec
(5MB/sec)
1MB/sec

14. AWS KINESIS STREAMS
KINESIS STREAMS - SHARD CALCULATION
▸ 10 Shards
▸ 10,000 records / sec
▸ 10MB / sec

15. DEMO

16. AWS KINESIS STREAMS
DEMO
▸ Create a stream with 1 shard
▸ Put 10 records / sec
▸ Read in batches every sec
{ RecordId: 11, KeyPressCount: 22, UserId: 33 }

17. KINESIS
FIREHOSE

18. AWS KINESIS FIREHOSE

19. AWS KINESIS FIREHOSE
TERMINOLOGY
▸ Firehose delivery stream
▸ record - 1MB max
▸ data producer
▸ buffer size, buffer interval

20. AWS KINESIS FIREHOSE
DATA DELIVERY
▸ KINESIS STREAM
▸ S3
▸ Redshift
▸ Elasticsearch

21. DEMO

22. KINESIS
ANALYTICS

23. AWS KINESIS ANALYTICS

24. AWS KINESIS ANALYTICS

25. AWS KINESIS ANALYTICS
TERMINOLOGY
▸ Input
▸ Application Code
▸ In-App-Streams
▸ Pumps
▸ Streaming SQL
▸ Output

26. AWS KINESIS ANALYTICS
STREAMING SQL
▸ Tumbling Window
[...] GROUP BY
FLOOR((“SOURCE_SQL_STREAM_001”.ROWTIME – TIMESTAMP
‘1970-01-01 00:00:00’) SECOND / 10 TO SECOND)
▸ Sliding Window
SELECT AVG(change) OVER W1 as avg_change
FROM "SOURCE_SQL_STREAM_001"
WINDOW W1 AS (PARTITION BY ticker_symbol RANGE INTERVAL
'10' SECOND PRECEDING)

27. AWS KINESIS ANALYTICS
STREAMING SQL - TUMBLING WINDOW

28. DEMO

29. AWS KINESIS ANALYTICS
DEMO
▸ Check BDMeetup Application in AWS Console
▸ Producer / Consumer
PRODUCER
APP
BDMEETUP STREAM
ANALYTICS
APP
BDMEETUP-OUTPUT
STREAM
CONSUMER
APP

30. THANK YOU