Trent Hornibrook gave a recent talk at the Infracoders meet-up playing a thought experiment with the audience on 'what would be your tech decisions if you were given a blank cheque at at startup'. Trent, recently working for a start-up then shared what decisions he made, and why

1. Greenfields tech
decisions

2. Me
• Trent Hornibrook
• Cevo
• Work to solve problems with great clients
• Like DiUS / Thoughtworks / Cogent
• but much much better

3. This talk
• Give me feedback on the tech choices you
would pick
• I will share with you what I picked and why

4. Imagine this:
Example startup
• They purchased some home grown software
• Need to deliver quickly

5. Day 1
• Two java apps iterated on for the past 10 years
• SQLSERVER datastores
• Think we may need to deploy on physical tin
• Does 80% of what we want from a business
perspective

6. The thing

7. Decisions
• You have carte blanche - what do you do?

8. Greenfields
• What would you choose:
• Team communication
• Hosting the code
• Build / packaging / where to deploy
• Logging / Monitoring / Alerting

9. Team communication
• What would you use?

10. Team communication
• Slack
• Hipchat
• Skype
• Hangouts
• Nothing

11. Team communication
• Slack
• Hipchat
• Skype
• Hangouts
• Nothing

12. Team communication
• That was easy

13. Hosting of code
• What would you use?

14. Hosting of code
• GitHub
• GitHub enterprise
• Bitbucket
• AWS CodeCommit
• GitLab
• Self rolled

15. Hosting of code
• GitHub
• GitHub enterprise
• Bitbucket
• AWS CodeCommit
• GitLab
• Self rolled

16. Hosting of code

17. Hosting of code
• Kinda hooks into the Atlasian suite
• Can pull commits into Jira stories

18. CI
• What would you use?

19. CI
• BuildKite
• Bamboo
• Jenkins
• TravisCi
• CircleCi
• Enterprise Jenkins

20. CI
• BuildKite
• Bamboo
• Jenkins
• TravisCi
• CircleCi
• Enterprise Jenkins

21. Buildkite

22. Buildkite

23. Buildkite

24. Buildkite

25. Buildkite

26. Buildkite

27. Buildkite

28. Buildkite
• Fully described pipelines as code in the code
• Manage nothing (well very little)
• Autoscale builds
• Great security pattern
• Local community
• Have you tried Bamboo?

29. Where to host
• Where would you host the app?
• Would you choose a IaaS platform or PaaS?
• What would you use?

30. Where to host
• IaaS
• AWS
• Azure
• Google Cloud
• PaaS
• Beanstalk
• Heroku

31. Where to host
• IaaS
• AWS
• Azure
• Google Cloud
• PaaS
• Beanstalk
• Heroku

32. AWS Account layout
• Account layout phase 1
• nonprod - staging
• prod - prod
• build - where we build & package - ci things
• audit - cloudtrail
• billing - consolidated billing

33. AWS Account layout

34. AWS Account layout

35. AWS Account layout

36. AWS Account layout
• Account layout phase 1
• nonprod - building & staging
• prod - prod
• master - saved package
• audit - cloudtrail
• billing - consolidated billing

37. Where to host
• The AWS horse is a safe bet
• Know we were going to grow and wanted more
control (IaaS over PaaS)
• Almost everyone knows AWS - skills transference
• Strong knowledge in good AWS security
• IAM / Security groups etc

38. AWS Account Layout
• Lay out everything the same
• Exceptions will occur, but don't start with them
• Seperate account by access control
• Avoid VPC peering!
• If you VPC peer, why have different accounts?

39. The Artefact / packaging
• What would you use?

40. The Artefact / packaging
• Docker container
• RPM / Deb
• Zip file
• Baking of AMIs / VMKDs
• rsync’ing of files

41. The Artefact / packaging
• Docker container
• RPM / Deb
• Zip file
• Baking of AMIs / VMKDs
• rsync’ing of files

42. The Artefact / packaging

43. The Artefact / packaging
• Hedged bets on needing to deploy onto
physical tin
• Like baking AMIs without the cost
• Helps (or should help) local development

44. Config as code
• What would you use?
• Caveat - it is just this for the moment:

45. Config as code
• Just cloudformation
• Ansible
• Chef
• Puppet
• Confluence doc

46. Config as code
• Just cloudformation
• Ansible
• Chef
• Puppet
• Confluence doc

47. Config as code

48. Config as code

49. Config as code
• Simplest solution
• Not complex enough to introduce Ansible /
Chef / Puppet / etc
• Not big enough for service discovery
• Most minimal repeatable solution

50. Logging
• What would you use?

51. Logging
• Splunk
• Logstash
• Loggly
• SumoLogic
• Cloudwatch Logs

52. Logging
• Splunk
• Logstash
• Loggly
• SumoLogic
• Cloudwatch Logs

53. Logging

54. Logging

55. Logging

56. Logging

57. Logging
• Prefer to pay for a SaaS provider for this
• You look after the logs, I’ll look after the app
• SumoLogic was Splunk like without $plunk
• Didn't spend a lot of time deciding - picked one
and moved on

58. Application monitoring
• What would you use?

59. Application monitoring
• NewRelic
• Cloudwatch
• Datadog
• Roll you own

60. Application monitoring
• NewRelic
• Cloudwatch
• Datadog
• Roll you own

61. What I start with
• Application performance (NewRelic)
• Mobile crash (NewRelic)
• Web endpoint from a third party (NewRelic)
• (Basic) Server metrics (NewRelic / Cloudwatch)
• Internal API HTTP endpoints (Cloudwatch)
• SQS queues (Cloudwatch)
• RDS disk usage (Cloudwatch)

62. What I avoid
• Application disk / CPU / memory usage (thats
not already free)
• Email queues
• MySQL ‘slave lag’

63. Alerting
• PagerDuty
• VictorOps

64. Monitoring / Alerting
• Pick a SaaS that works at an affordable price
• Don't spend too much time picking one
• Refine overtime once the app is up and running

65. The second month
• The app does 80% of what we want from a
business perspective
• Lets get a team together to work on the 20%
• Old-code-phobia
• Microservices-all-the-things

66. Config as code
• What would you use?
• Caveat - it is just this for the moment:

67. Config as code
• What would you use?
• Caveat - it’s this:

68. Config as code
Three approaches spring to mind:
• Extend the one ec2 instance docker run
approach
• ‘docker-compose’ on the ec2 instance
• Docker Scheduler

69. What I like
• Awesome security
• Tight IAM policies
• Security groups between the things
• Write once, deploy everywhere

70. Schedulers
• Kubernetes
• Mesos
• Swarm
• Rancher
• ECS

71. Schedulers
• Kubernetes
• Mesos
• Swarm
• Rancher
• ECS

72. Rancher
• Really easy to visualise what is going on
• Obtain access to the console logs
• Even run commands on the console!
• Can be controlled by docker-compose (plus a
rancher-compose file)
• Local dev++

73. Rancher

74. Rancher

75. Rancher

76. Rancher

77. Rancher

78. Rancher

79. Rancher

80. Rancher - pro
• Developer friendly
• docker-compose

81. Rancher - neg
• Doesn't fit well with IAM
• It looses its shit when ec2 hosts go away
• ECR container support is flimsy
• No scaling support
• Hard to be half in AWS (eg RDS) & Rancher

82. ECS

83. ECS
• Have to duplicate work
• Task definitions are not docker-compose files
• Multiple state machines
• Configuration?

84. ECS

85. ECS

86. ECS
• When a new container is built you just want to
update the single task definition

87. ECS

88. ECS
• But what if the infrastructure changes
• New / Change to RDS
• S3
• IAM policy

89. ECS

90. ECS - pro
• In the AWS ecosystem
• Surely ECS will get better, no?
• Task based IAM

91. ECS - neg
• Have to duplicate docker-compose into Task Definitions
• How you control the state
• Use ecs-cli ? Use ecs-deploy ? Use cloudformation?
• You MUST have log aggregation
• The UI is messy
• Hard to diagnose what is happening
• Task based IAM has wedged us - maturity?

92. Lessons
• Try and pick tools that have strong community
support & have common knowledge in the community
• Pick the solution that your broad team members
knows about
• Config-as-code all-the-things
• git blame away from understanding why
• Start simple and iterate