Recommended
More Related Content
What's hot
What's hot (20)
Similar to Ruby Microservices with RabbitMQ
Similar to Ruby Microservices with RabbitMQ (20)
Recently uploaded
Recently uploaded (20)
Ruby Microservices with RabbitMQ
- 2. Content • Intro from monolith decoupling to microservices integration • The Main Part microservices messaging with RabbitMQ/AMQP • Ruby Code Refactoring from ActiveRecord Callback to Consumer as a Microservice • A Simple Microservice Deployment in docker container to AWS ECS
- 3. Let's start with a bad example
- 4. A better example ... or can we just publish the event and let subscribed consumers do the work? a complete to-do list for this event
- 5. Decoupling Publisher / Producer Subscriber / Consumer Subscriber / Consumer Subscriber / Consumer
- 6. Monolith vs Microservices vs Complexity in Interactions Code Complexity
- 7. Microservice • a small program that handle one task • independently deployable • work together by communicating so that can accomplish the larger task • integrated system on the whole provides value https://martinfowler.com/ articles/microservices.html https://youtu.be/wgdBVIX9ifA
- 8. App Integration Options • File Transfer • Shared Database • Remote Procedure Invocation • Messaging While all four approaches solve essentially the same problem, each style has its distinct advantages and disadvantages. http://www.enterpriseintegrationpatterns.com/patterns/messaging/IntegrationStylesIntro.html
- 9. File Transfer Consumer A Consumer B Consumer C DATA FILE Producer Export Import
- 10. Shared Database Consumer A Consumer B Consumer C Producer DB
- 11. Remote Procedure Invocation Consumer A Consumer B Consumer C Producer Call Result
- 12. Messaging Exchanges Bindings Queues Consumer A Consumer B Consumer C Producer AMQP 0-9-1 Message Broker
- 13. • a binary, networking protocol with minimal overhead • originated in 2003 by John O'Hara at JPMorgan Chase • version 0.9.1 was specified in 2008 • in 2011 OASIS standards group change it to a significantly different beast
- 14. AMQP 0-9-1 • virtual host is a logical partition within the server (a multi-tenant system similar to virtual hosts in Apache/Nginx) • each of virtual hosts can have many connections, channels, queues, exchanges and some other things • bindings binds an exchange to a queue or many queues • exchange can be: direct, fanout, with topic or headers • queue is a buffer that holds messages on behalf of a consumer (or consumers)
- 15. The Producer • external application which creates messages and decides: • how the attributes should be configured for routing • from which exchange the messages should start from • what is the actual payload that is being sent
- 16. The Message • an atomic unit of processing • consists of: • content (body, bare message or payload) • attributes - metadata about the message like content type, encoding, routing key, whether the message will be persistent or not, and whether it has a priority level, etc. • is created (published) by the producer • may go through more than one exchange before landing in the right queue
- 17. The Exchange • Direct exchange route messages based on an exact match with the specified routing key • Fanout exchange automatically route the message to all the queues known to them (ignores the routing key) • Topic exchange pattern match on the routing key (topic) to route the messages • Header exchange use the message header attributes for matching the queue • Default (anonymous) exchange is a direct exchange (created automatically) which routes messages with empty exchange name - it compares routing key with the queue name
- 18. Multiple Queues and Consumers Consumer A Queue 1 Queue 2 Queue 3 Consumer B Consumer C
- 19. The Consumer • external application which is subscribed to one or more queues • alerted whenever a message shows up in the subscribed queue • can poll the queue at regular intervals to see which messages were added in the queue since the last time it made the request • can send acknowledgments back to RabbitMQ that the message has been: • received (known as ack), or • rejected (nack for negative acknowledgments)
- 20. Bindings • Routing Key • Publisher: exchange.publish(payload, routing_key: 'foo') • Queue Binding: queue.bind(exchange, routing_key: 'foo') • Headers • Publisher: exchange.publish(payload, headers: { ... }) • Queue Binding: queue.bind(exchange, arguments: { ... })
- 21. • an message broker implemented with Erlang/OTP • implements all the AMQP 0.9.1 concepts: messages, queues, exchanges, bindings, virtual hosts ... • ... and with plugins for other messaging protocols such as STOMP, XMPP, and more • has fantastic client support in a variety of popular languages: Ruby, Python, Java, PHP, C#, JavaScript, Go, Elixir, Objective-C, Swift, ...
- 22. Gems • Bunny - AMPQ client for Ruby (MRI) • March Hare - AMPQ client for JRuby • Sneakers - a background consumer/worker • Hutch - asynchronous inter-service communication • RabbitMQ HTTP API client (various management features) • Ruby RabbitMQ Clients Blog: http:// blog.rubyrabbitmq.info/
- 23. Features • No message loss • Persistent Messages • Publisher Confirms • Message Acknowledgment • Mirrored Queues • Message Ordering • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs • Consumer Priorities • Federation ... and many more
- 24. Back to Refactoring ... or can we just publish the event and let subscribed consumers do the work?
- 26. Publish Message with Topic
- 27. Message Consumer
- 29. Message Consumer
- 32. Messaging Patterns (65) Creative Commons Attribution 4.0 license.http://www.enterpriseintegrationpatterns.com/patterns/messaging/index.html
- 33. Microservice Messaging Benefits • language-agnostic • amplifies loose-coupling • makes scaling and rearchitecting simpler • better reliability and availability - messages will be waiting until the consumer is ready to process them • multiple communication patterns: events, message/reply, notification, async request-response, pub/sub, etc.
- 34. Microservice Messaging Drawbacks • introducing new complexity into the system (the message broker) • a failure in the message broker can cause severe effects on the system (highly-available message broker) • big messages can cause network congestion • "eventual data consistency" (instead of immediate consistency across different microservices)
- 35. Message Consumer Microservice Deployment with Docker to AWS ECS
- 36. Installing Docker CE on Workstation • macOS: https://store.docker.com/editions/community/docker-ce- desktop-mac • Linux Ubuntu: https://store.docker.com/editions/community/docker-ce- server-ubuntu • or any other Linux distro: https://store.docker.com/search? offering=community&operating_system=linux&type=edition
- 37. My Ruby Project Structure
- 38. Dockerfile
- 40. Elastic Container Registry export repositoryName=my_consumer export repositoryURI=123456789012.dkr.ecr.us-east-1.amazonaws.com export containerName=my_consumer export ver=1.0.0 export imageURI=$repositoryURI/$repositoryName:$ver aws ecr create-repository --repository-name $repositoryName eval $(aws ecr get-login --no-include-email) docker tag $containerName:$ver $imageURI docker push $imageURI
- 41. Elastic Container Service (ECS) • logical way to group resources (Tasks and Services) • currently provides two launch types: • EC2 • Fargate (abstract away EC2 instances) collection of ECS resources: https://github.com/nathanpeck/awesome-ecs export clusterName=staging aws ecs create-cluster --cluster-name $clusterName
- 42. ECS Cluster Building Blocks Task Definition • specify the resources for a Docker container or group of containers, such as: docker image (registry), ports, CPU/RAM, logs, any volumes, environment variables, etc. Task • running containers according to the Task Definition (one-off or long-running) • TaskRole allow access to S3, DynamoDB, etc. Service • manage long-running tasks (defines desired count and replace failed containers) • integrates with Elastic Load Balancer
- 43. ECS Task Definition { "containerDefinitions": [ { "command": [ "bin/worker" ], "workingDirectory": "/app", "essential": true, "image": $imageURI, "logConfiguration": { "logDriver": "awslogs" }, "name": $containerName } ], "cpu": "256", "memory": "512", "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole", "family": "my-consumer-fargate", "networkMode": "awsvpc", "requiresCompatibilities": [ "FARGATE" ] }
- 44. ECS Task export taskDefinition=my-consumer-fargate aws ecs register-task-definition --cli-input-json file://my-consumer-fargate-task-definition.json aws ecs run-task --launch-type FARGATE --cluster $clusterName --task-definition $taskDefinition --network-configuration "awsvpcConfiguration={subnets=[subnet-abc], securityGroups=[sg-01]}"
- 45. ECS Service • runs and maintains the required number of tasks associated with the elastic load balancer aws ecs create-service --cluster $clusterName --service-name my-consumer-service --task-definition $taskDefinition --desired-count 2 --launch-type FARGATE --network-configuration "awsvpcConfiguration={subnets=[subnet-abc], securityGroups=[sg-01]}"
- 46. Deploying Updates 1. Build a new image and push it to the repository 2. Create a new revision of the Task Definition (revision numbers increment automatically) 3. Update Service to use new Task Definition revision or simply use: https://github.com/silinternational/ecs-deploy ecs-deploy -c $clusterName -n my-consumer-service -i $imageURI
- 47. Advanced Example of AWS ECS Deployment with Terraform VPC with 2 subnets (1 public and 1 private) in each Availability Zone https://github.com/duduribeiro/terraform_ecs_fargate_example https://thecode.pub/easy-deploy-your-docker-applications-to-aws-using-ecs-and-fargate-a988a1cc842f
- 48. AWS CodeBuild AWS CodePipeline screenshots from: https://thecode.pub/easy-deploy-your-docker-applications-to-aws-using-ecs-and-fargate-a988a1cc842f
- 49. AWS ECS vs Kubernetes 1. AWS ECS can not be run on-premise 2. AWS ECS lacks advanced features These two differences can either be seen as weakness or as strengths.