We tell the story of how we went from Heroku to Amazon Web Services. Including scary performance charts ;)

1. From Heroku to AWS
a Migration Story

2. Who am I?
• Sebastian Schleicher, Head of IT @Blinkist
http://about.me/sebastian.schleicher
• Blinkist
iOS & Android app with some backend & web app,
Head count of 40 with 10 Developers, 1 DevOps (since 2016)

3. The App in Focus
Ruby API
MongoDB
ENV["MONGODB_URL"]

4. Let’s get back to the
beginnings…

5. The Beginnings (2012)
• Dedicated Server, 79€/Month
• Virtualized Stack
• Puppet or Chef, don't remember
• Java API
• Subversion
Set up by Student Friend
}

6. Deployment Process
build *.war file
ftp upload api.war > webapps/
Tomcat auto deploy ✅
❓
tail -f tomcat.logs
ssh tomcat-machine

7. Deployment Process
• ❌ Manual upload of artifact files
• ❌ No tests
• ❌ Unclear deployment state
• ❌ Some Tomcat instance ….
• 🚙 Subversion
• ✅ Centralized logging
(as there’s only one server 🙈)

8. Problems
• Our experience
We - naive developers - thought that’s a good way of doing it
• No knowledge of infrastructure
We’re good application developers, no more
• Hardware outtake
During final presentation in front of our investors. We had to recover
somehow without the student friend 😱
We’ve had to switch, Heroku comes to our rescue

9. Heroku
• Developers must love it
• Simple Deployment via Git remote
• No operations
• No infrastructure
• Beautiful CLI
• Scale at ease

10. Blinkist’s Heroku Stack
• Switched from Java to Ruby and started writing tests
• Switched from Subversion to Git/-hub
• CI/CD with Codeship
• Logging via Heroku Log Drains to Loggly
Set up within 1 week! 😂

11. And we scaled it.

12. Heroku App Count
0
4
8
12
16
2012 2013 2014 2015 2016
15
7
4
3
2

13. Deployment Process
git push
test
deploy
✅
@Codeship
@Heroku

14. Deployment Process
• 🚀 Git with GitHub
• ✅ Everything’s tested before …
• ✅ … CI tool automatically deploys
• ✅ Centralized logging (also more multiple apps)
We thought to be heroes for ever and ever!

15. Advantages
• No 24/7 availability
We could go into our weekend within fearing for an outtake
• No maintenance
Heroku takes care of patching systems and keeping everything
healthy
• New features every month
Heroku continuously adds new features like Review-Dynos (swag!)
This is the best of the best! 😻

16. but…

18. Disadvantages
• Availability
Heroku dynos are-unreachable quite frequently for different reasons
• Multi-tenant system
One single Heroku Customer and screw the stack
(our dyno’s IPs got blocked a few times by CDNs)
• Low network performance
Inbound and outbound latency can be pretty high
• Request queuing
Takes up to 10% of the overall request time
• Performance improvements are pricey
2015 Heroku introduced new, more expensive dyno types

19. Disadvantage: Lock-in
• Lack of infrastructure knowledge 💣
Heroku let us scale without learning about infrastructure
• No fallback
We weren’t able to point our DNS to another system
• Poor performance
We couldn’t optimize our code anymore regarding speed

20. We wanted to
• Make our apps deployable everywhere (pot.)
No more platform lock-in
• Gain more control over the infrastructure
Change network, memory, CPU to our needs
• Keep costs at an adequate level
We need to grow reasonable
• Keep our current release pace and deployment quality
The new solution has to be simple to work with
• Teach developers to work with infrastructure
Empower them to master the whole stack

21. We decided to
dockerize and AWSify
our stack …

22. … and hired a
DevOps Hero.

23. The goal for deployment
git push
test
deploy
✅
@Codeship
@AWS

24. Our approach
• Dockerize the service
We need to have a containerized runtime env
• Deploy via Elastic Beanstalk (Multicontainer)
Create a Heroku-style deployment experience
• CI/CD via Codeship
Using their script-based integration

25. First challenge from Business
• World wide App feature in October 2016
In Apple AppStore and Google PlayStore
• Potentially high increase in load
From ~2.5k RPM to 15k (6x!)

26. We dockerized and deployed
to Elastic Beanstalk in high
speed mode

27. …and benchmarked
this setup with locust.io

28. AWS
6x c4.xlarge

29. Heroku

30. That’s impressive
(and scary)

31. We switched the
API to run on AWS

32. Heroku shutdown

33. AWS startup

34. AWS API today

35. We can now challenge
any App feature

36. But what about
deployment?

37. Beanstalk
Deployment Process
git push
docker build
docker test
✅
Collect and prepare
Beanstalk config files in
BusyBox
zip -r config.zip *
upload to S3
Create new app
version for ENV
Wait for Beanstalk
to become green
@CI/CD

38. Deployment Duration
Minutes
0
4
8
12
16
Heroku Beanstalk via Codeship Beanstalk via CircleCI
10
15
2

39. Timeout during deployment
Due to Elastic Beanstalk, not CircleCI

40. Deployment Process
• ✅ Everything automated
• ❌ Random test failures
(e.g. DB wasn’t ready when specs started)
• ❌ Custom-build scripts
• ❌ Long running deployment process
• ❌ Rolling updates sometimes fail on single machines
This felt like a big step backwards

41. Side note: Update Env Variable
Minutes
0
2,5
5
7,5
10
Heroku Beanstalk
10
1

42. And the
developer experience?

43. Developer experience
• Annoyance due to deployment times
Our devs get nervous if a deployment takes more
than 10 minutes
• Lack of simple CLI
They need to ssh into the machines and navigate
around with root permissions
• Lack of some Heroku features
Like the (swag!) PR review apps

44. Let’s check costs!

45. Beanstalk setup per app
Load Balancer
EC2 EC2 EC2
API

46. Beanstalk setup at scale
ELB
EC2 EC2 EC2
ELB
EC2 EC2
ELB
EC2 EC2 EC2
API
ELB
EC2 EC2 EC2
WEB
SERVICES
ELB
EC2 EC2 EC2
Already 14 EC2 plus 5 ELB instances in this example

47. Beanstalk setup
• Pricy scaling
Having one EC2 instance per docker process is
extremely expensive with multiple apps/services.
Usually you’ll have at least 1 ELB plus 2 EC2 instances
• Hard to maintain
Changes to the infrastructure need to be rolled out
across apps
• Increasing deployment time
The more instances we have per ELB, the longer the
deployment takes

48. So where are we now?

49. Recap: We wanted to
• ✅ Make our apps deployable everywhere
Thanks to Docker, potentially easy
• ✅ Gain more control over the infrastructure
Done, we can choose the EC2 instance type easily
• ❌ Keep costs at an adequate level
Beanstalk scales at high prices as many EC2 instances are needed
• ❌ Keep our current release pace and deployment quality
The new process is error prone and slow
• ❌ Teach developers to work with infrastructure
They avoid working in this infrastructure as it’s too risky to f*** it up

50. How can we improve?

51. Costs: Beanstalk setup
Classic Load Balancer
EC2 EC2 EC2
> Memo: We’re talking about Multicontainer environment <
API

52. Classic Load Balancer
ECS ECS ECS
EC2 EC2 EC2
What is powering it?
API

53. Multiple Apps
Classic Load Balancer
ECS ECS ECS
API
Classic Load Balancer
ECS ECS ECS
Service A
EC2 EC2 EC2 EC2 EC2 EC2

54. So: Switch to pure ECS
Application Load Balancer
API
Service A
Service B
API
Service A
Service B
API
Service A
Service B
ECS ECS ECS
EC2 EC2 EC2
us-east1-a us-east1-b us-east1-c

55. Release pace and quality
• Split Docker images
Pre-create docker base images with GEMs installed
• Minimize Docker images
We continuously evaluate other ways to minimize image
sizes, e.g. docker-squash, clean up GEMs
• Powerful ECS instances over small Beanstalk EC2s
Bigger instances with higher networking will deploy docker
images quicker

56. Developer satisfaction
• Mistrust in Docker for development
Our developers are hard to convince of the benefits of Docker
for local development - never change a running system…
• Workshop for infrastructure basics
We’re continuously holding workshops on Docker for
development and other infrastructure topics
• Stabilize releases
Only a trusted and reliable system will be accepted
• Build a CLI
We want to build or find a CLI to easily interact with our apps

57. Maintainability of infrastructure
• No more UI based infrastructure
Our current AWS services are created fully by click
• Scripted infrastructure
We believe that every bit of our tech stack has to be
code. Terraform seems to be a promising solution
that offers a descriptive approach to creating a
reproducible, reviewable and testable environment
of infrastructure components and there their
interconnections.
http://www.meetup.com/de-DE/terraform-berlin-user-group/

58. We’ll talk about our
results and learnings
in Q1/2017

59. From Heroku to AWS
a Migration Story
to be continued

60. I’m looking forward to
receive your feedback
and inspiration
- Thank you -