Shows how to build a distributed sy,stem using Vertx, Camel, AWS Lambda, Kinesis and Kafka

1. Sascha Möllering | zanox AG
Event-driven architecture with Java

2. Sascha Möllering
sascha@autoscaling.io
Lead Engineer at zanox AG
http://autoscaling.io
@sascha242
About me

3. Agenda
• Basics
• Infrastructure
• Software & Frameworks
• Deployment
• Putting it all together
• Q&A

4. Agenda
• Basics
• Infrastructure
• Software & Frameworks
• Deployment
• Putting it all together
• Q&A

5. Basics
• Today: building event driven system
• Using:
– Apache Kafka/Amazon Kinesis
– Docker
– Vert.x
– Apache Camel/AWS Lamdba
– Google’s Protobuf

6. Basics

7. Basics
• Producers putting data into messaging
system
• Messages in Google’s Protobuf format
• Consumers pulling data from
messaging system

8. Agenda
• Basics
• Infrastructure
• Software & Frameworks
• Deployment
• Putting it all together
• Q&A

10. Infrastructure
• Publish-subscribe messaging
• Implemented as distributed commit log
• Fast: 100s of MB (reads and writes) per
s from thousands of clients
• Scalable: elastically and transparently
expanded without downtime
• Durable: Messages persisted on disk

11. Infrastructure

12. Infrastructure

13. Infrastructure
• Depends on Apache Zookeeper
• Zookeeper for:
– Cluster management
– Offset storage

15. Infrastructure
• Managed service for real-time data
processing
• Decoupling services
• Data stored for 24 hours
• 1MB messages max

17. Infrastructure
• Stream:
– Ordered sequence of data records
– Data records are distributed into shards

18. Infrastructure
• Shard:
– Group of data records in a stream
– 1MB write per second
– 2MB read per second
– 1000 puts per second

20. Infrastructure
• Package application with dependencies
• Standardized unit for software
development
• Layered filesystem, share common files
• Isolate applications from each other

22. Infrastructure
• Docker container: stripped-to-basics
version of a Linux operating system
• Docker image: software you load into a
container

23. Infrastructure
• Docker image built with a Dockerfile
• Docker images built using “inheritance”
• Custom image based on “base image”

24. Agenda
• Basics
• Infrastructure
• Software & Frameworks
• Deployment
• Putting it all together
• Q&A

26. Software & Frameworks
• Toolkit for reactive applications
• Based on the JVM
• Event driven and non-blocking
• Polyglot (Java, JS, Groovy, Ruby)
• Lightweight and modular

27. Software & Frameworks

28. Software & Frameworks
• Producer application built using Vert.x
• Sends a message every 5s
• Kafka or Kinesis depending on
deployment target

29. Software & Frameworks
But … how can we
detect where the
application is running?

30. Software & Frameworks
http://169.254.169.254/latest/meta-data

32. Software & Frameworks
• Framework based on EI Patterns
• Small library with minimal dependencies
• Define routing and mediation rules

34. Software & Frameworks

36. Software & Frameworks
• Language-neutral
• Platform-neutral
• Extensible mechanism for serializing
structured data
• Support for Java, Python, and C++

37. Software & Frameworks

38. Software & Frameworks

39. Software & Frameworks

40. Software & Frameworks

42. Software & Frameworks
• Compute service
• Runs your code in response to events
• Manages underlying compute resources

43. Software & Frameworks
• Triggered by:
– Modifications in S3 buckets
– Notifications by SNS
– Messages in Kinesis
– Table updates in DynamoDB

44. Software & Frameworks
• Code run in Lambda: “Lambda function”
• Don’t confuse with Java 8 Lambda
• Lambda functions support
– Java 8
– JavaScript

45. Software & Frameworks
• Two different event models
– Pull event model
– Push event model

46. Software & Frameworks

47. Software & Frameworks

48. Software & Frameworks

49. Software & Frameworks
• Building the applications
– ingestion-service
– kafka-consumer-camel
– kinesis-consumer-lambda

50. Software & Frameworks
• ingestion-service
1. git clone
https://github.com/SaschaMoelle
ring/ingestion-service.git
2. mvn -Dmaven.test.skip=true
package
3. docker build -t
autoscaling/ingestion-service .

51. Software & Frameworks
• kafka-consumer-camel
1. git clone
https://github.com/SaschaMoelle
ring/kafka-consumer-camel.git
2. mvn -Dmaven.test.skip=true
package
3. docker build -t
autoscaling/kafka-consumer .

52. Software & Frameworks
• kinesis-consumer-lambda
1. git clone
https://github.com/SaschaMoelle
ring/kinesis-consumer-
lambda.git
2. mvn -Dmaven.test.skip=true
package

53. Agenda
• Basics
• Infrastructure
• Software & Frameworks
• Deployment
• Putting it all together
• Q&A

55. Deployment
• Locally
1. Start the Spotify Kafka Docker Container

56. Deployment
• Why Spotify Kafka Docker Image?
– Kafka depends on Zookeeper
– Spotify Kafka runs Kafka and Zookeeper
– No dependency to external Zookeeper
– Runs out of the box

57. Deployment
• Locally
2. Start the Apache Camel Kafka consumer

58. Deployment
• Kafka-Consumer Docker Container
– Based on phusion/baseimage
– Installs Oracle Java 8
– Add consumer Fat-JAR
– Starts the Fat-JAR

59. Deployment
• Locally
3. Start the Vert.x Kafka Producer

60. Deployment
• Vert.x Producer Docker Container
– Based on phusion/baseimage
– Installs Oracle Java 8
– Add producer Fat-JAR
– Starts the Fat-JAR

61. Deployment
• Requirements for AWS:
– VPC
– User Role for Kinesis access from EC2
– User Role for Kinesis access from Lambda
– EC2 instance
– Kinesis stream
– Lambda package

62. Deployment
• In AWS
– Create Kinesis stream (in our case
SUMMIT_STREAM)

64. Deployment
• In AWS
– Create Lambda function
• Upload JAR to S3 bucket
• Specify function
• Add event source (SUMMIT_STREAM)

67. Deployment
• In AWS
– Start an EC2 instance
• t2.small is sufficient
• Install Docker and run container using EC2
user data
• Important: select correct IAM role

68. Deployment
• EC2 User Data
#!/bin/bash -ex
yum -y update
yum install docker -y
service docker start
docker run autoscaling/ingestion-service

71. Agenda
• Basics
• Infrastructure
• Software & Frameworks
• Deployment
• Putting it all together
• Q&A

72. Putting it all together
• Integration of Kinesis and Kafka
– Kinesis consumer that processes records
– Record processing -> sending to Kafka
– AWS Lambda perfect choice
– Problem: Lambda and VPN (VPC) not
working*

73. Putting it all together
• Integration testing Kinesis and Kafka
– AWS API:
• Create Kinesis stream in @BeforeClass
• Produce data and write into stream
• Delete stream in @AfterClass

74. Putting it all together

75. Putting it all together

76. Putting it all together
• Integration testing Kinesis and Kafka
– Spotify Docker Client
• Run Spotify Kafka container in @BeforeClass
• Produce data and write into stream
• Stop Spotify Kafka container in @AfterClass

77. Putting it all together

78. Putting it all together
• Integrationtests Kinesis and Kafka
– Put messages into Kinesis
– Consumer messages in application
– Put messages in Kafka
– Consume messages from Kafka
– Compare messages

79. Putting it all together
• Integrationtests Kinesis and Kafka
– After tests: clean up infrastructure
– Very cost effective
– Real world tests without mocking
– Quite fast

80. Recap
• What have we achieved today?
– We created a distributed, message driven
system
– Based on JVM and Docker
– Running locally and AWS

81. Q&A

82. Resources
• ingestion-service
– https://github.com/SaschaMoellering/ingest
ion-service.git
– https://hub.docker.com/r/autoscaling/ingest
ion-service/

83. Resources
• kafka-consumer
– https://github.com/SaschaMoellering/kafka-
consumer-camel
– https://hub.docker.com/r/autoscaling/kafka-
consumer/

84. Resources
• EC2 User Data
– https://gist.github.com/SaschaMoellering/c
6ee24ec999325c43e90
• EC2 User Role
– https://gist.github.com/SaschaMoellering/
a971fb73626f41ad80f4

85. Resources
• Lambda User Role
– https://gist.github.com/SaschaMoellering/b
14540b144263e5fea4b

86. Resources
• kinesis-consumer-lamdba
• https://github.com/SaschaMoellering/kinesis-
consumer-lambda

87. Resources
• Spotify Kafka Docker Image
– https://github.com/spotify/docker-kafka.git
– https://hub.docker.com/r/spotify/kafka/
• Spotify Docker Client
– https://github.com/spotify/docker-client.git