How to fan-out Amazon Kinesis Streams with Amazon Kinesis Analytics to solve read throughput and duplicated records issues. Longer article here: https://www.linkedin.com/pulse/how-fan-out-amazon-kinesis-streams-alex-casalboni GitHub project here: https://github.com/alexcasalboni/kinesis-streams-fan-out-kinesis-analytics

1. clda.co/cloudcamp-­‐kinesis
A Serverless implementa7on of
Kinesis Streams fan-­‐out
(w/o AWS Lambda)
07/06/2017

2. About Me
@alex_casalboni
clda.co/cloudcamp-­‐kinesis
Computer Science Background
Sr. SoCware Engineer & Web Developer
Cloud Evangelist @ Cloud Academy
Serverless Italy Meetup Co-­‐Organizer

3. Amazon Kinesis Streams
Managed real-­‐7me streaming
Streams as data buffers
Data reten7on (up to 7 days)
No built-­‐in autoscaling
Records distributed into Shards
Read/write throughput (per shard)
clda.co/cloudcamp-­‐kinesis

4. clda.co/cloudcamp-­‐kinesis

5. clda.co/cloudcamp-­‐kinesis

6. clda.co/cloudcamp-­‐kinesis

7. Kinesis Streams Fan-­‐out
AWS Lambda Amazon Kinesis Analy3cs
Duplicated records?
Batch sizing?
Real-­‐7me SQL
Mul7ple output Streams
github.com/awslabs/aws-lambda-fanout clda.co/kinesis-analytics-fanout
clda.co/cloudcamp-­‐kinesis

8. Kinesis Analy@cs real-­‐@me SQL
clda.co/cloudcamp-­‐kinesis

9. clda.co/cloudcamp-­‐kinesis

10. Gotchas
No mul7ple outputs in web console
It’s not cheaper (more shards)
Kinesis Analy7cs pricing is weird
No CloudForma7on support (yet)
Records data model must be defined
SpliUng with WHERE condi7ons
clda.co/cloudcamp-­‐kinesis
(hourly rate for each KPUs)

11. SpliFng a fat stream with SQL condi@ons
clda.co/cloudcamp-­‐kinesis

12. Learn more
clda.co/cloudcamp-­‐kinesis
docs.aws.amazon.com/kinesisanalytics/latest/dev/streaming-sql-concepts.html
github.com/alexcasalboni/kinesis-streams-fan-out-kinesis-analytics
linkedin.com/pulse/how-fan-out-amazon-kinesis-streams-alex-casalboni
@alex_casalboni

13. Thank you =)
07/06/2017 clda.co/cloudcamp-­‐kinesis