Microservice platforms are finally becoming a reality: Mesos, Kubernetes, and a whole bunch of PaaS-style offerings are available, but the reality is that these platforms still don’t provide everything you need in order to build a fully functional microservice ecosystem. Come to this session to learn about the essential deployment, orchestration, and glue components that often have to be self-assembled. The presentation begins by looking at deployment techniques and tools and examines where to test (QA, staging, or production), how to test (integration and contracts), and how to separate deployment and release. It then discusses orchestration, configuration, and service discovery. Finally it looks at essential glue such as logging, monitoring, and alerting.

1. Building a microservice ecosystem
Daniel Bryant
@danielbryantuk
OpencRedo

2. Today
• What is a ‘microservice ecosystem’?
• Local dev of services becomes challenging
• Think about your build pipeline(s)
• Testing requires a paradigm shift
• Push to production as early as possible
24/11/2016 @danielbryantuk

3. @danielbryantuk
• Chief Scientist at OpenCredo, CTO at SpectoLabs
ü Transforming organisations through technology and teams
ü Agile, Lean, Architecture, CI/CD, DevOps
ü Microservices, cloud, Containers, Java, Go, Docker, Kubernetes
• London Java Community Associate
• Adopt OpenJDK and JSR
• InfoQ Editor, DZone MVB, VOXXED, O'Reilly
24/11/2016 @danielbryantuk

4. My Biggest Claim to Fame in Microservices?
24/11/2016 @danielbryantuk

5. Seriously Though...…
24/11/2016 @danielbryantuk
skillsmatter.com/skillscasts/7004-our-journey-to-world-gifting-
domination-how-notonthehighstreet-com-embraced-docker

6. So, What is a microservice?
“Loosely coupled service oriented architecture
with bounded contexts”
Adrian Cockcroft
“Applications that fit in your head”
James Lewis
24/11/2016 @danielbryantuk

7. Why? The move towards Cloud-native
• Hypothesis-driven business (and development)
• Microservice architecture (and 'Micro' teams)
• Devops mindset/culture (shared, learning/feedback & Mechanical sympathy)
• Continuous delivery
• Automated, self-service platform
Hat tip @caseywest
24/11/2016 @danielbryantuk

8. Exploring the Ecosystem
24/11/2016 @danielbryantuk

9. Adrian Cockcroft’s Thoughts
24/11/2016 @danielbryantuk
www.slideshare.net/adriancockcroft/microxchg-microservices

10. Alexis Richardson’s Thoughts
24/11/2016 @danielbryantuk
gotocon.com/goto-london-2015/#!#schedulePopupExtras-7011

11. wikibon.com/wp-content/uploads/container_implementations.png
Technology Choices
24/11/2016 @danielbryantuk

12. 24/11/2016

13. What do I mean by ‘ecosystem’?
• Build
– Local development, pipelines and integration
• Test
– From local integration to End-to-end
• Deploy
• Operate
• Observe
– Monitoring/logging/alerting
24/11/2016 @danielbryantuk

14. Don’t forget about (RE)ArchitecturE Complexity
24/11/2016 @danielbryantuk

15. 24/11/2016 @danielbryantuk
www.infoq.com/news/2015/04/raffi-krikorian-rearchitecting

16. What do I mean by ‘ecosystem’?
• Build
– Local development, pipelines and integration
• Test
– From local integration to End-to-end
• Deploy
• Operate
• Observe
– Monitoring/logging/alerting
24/11/2016 @danielbryantuk

17. Build
24/11/2016 @danielbryantuk

18. Developing Locally: The Basics
• GitHub’s Boxen (Puppet)
• Pivotal’s Sprout (Chef)
• Mac-dev-playbook (Ansible)
• Hashicorp Vagrant
• Docker Compose (Docker machine/native)
24/11/2016 @danielbryantuk

19. Developing Locally
• The naive approach
– Replication of env per service
– …and dependencies and data stores and...
– Soon gets crazy
• Local profiles + mocking/stubbing
– Spring profiles + Mockito etc
24/11/2016 @danielbryantuk

20. Developing Locally
• Service virtualisation
– Hoverfly, Mountebank, Wiremock (Saboteur)
• ‘Production-in-a-box’ (IFTTT)
– Docker Compose, Vagrant, cf_nise_installer
• Environment leasing
– Create your own env (e.g. Hailo)
24/11/2016 @danielbryantuk

21. Developing Locally
www.opencredo.com/2015/09/20/working-locally-with-microservices/
24/11/2016 @danielbryantuk

22. Create a Pipeline
24/11/2016 @danielbryantuk

23. Key lesson learned (the Hard Way)
• Push through to production as early as possible!
• Detect and fix issues
– Technology
– Process
– People
24/11/2016 @danielbryantuk

24. Test
24/11/2016 @danielbryantuk

25. Always Remember...
24/11/2016 @danielbryantuk
blog.bbv.ch/2012/06/13/acceptance-test-driven-development/

26. Microservice Testing Basics
• Toby Clemson’s article
martinfowler.com/articles/microservice-testing
24/11/2016 @danielbryantuk

27. Microservice Testing Basics
• My article
https://www.specto.io/blog/recipe-for-designing-building-
testing-microservices.html
24/11/2016 @danielbryantuk

28. (multiple) Pipelines
• Big-bang release
– Beware of rubber stamping (distributed monolith)
– ‘semver’ if you must (semver.org) e.g. 1.2.1
• Gated release
– Critical path testing in Stage
• Single service Continuous delivery
– Consumer-based contracts
– Backwards compatibility
24/11/2016 @danielbryantuk

29. Service-level Integration Testing
• Contracts
– Pact-JVM github.com/DiUS/pact-jvm
– PACT broker
github.com/bethesque/pact_broker
– Examples:
github.com/mstine/microservices-pact
24/11/2016 @danielbryantuk

30. My Opinions
• BDD critical paths throughout application
– Including API journey
• Contract tests (failure is a conversation)
• BDD services API (e.g. Serenity BDD)
• Component test and unit test (as normal)
– Maven surefire/failsafe
24/11/2016 @danielbryantuk

31. Final Words on Testing
• Don’t forget the ‘ilities”
• Security / reliability
– ZAP (from the OWASP team)
– github.com/continuumsecurity/bdd-security
• Performance / scalability
– Jmeter (Jenkins Performance plugin)
– Gatling
– flood.io
24/11/2016 @danielbryantuk

32. Deploy
24/11/2016 @danielbryantuk

33. Separate Deploy and Release
• Feature flags
– Difficult at scale (and distribution)
– Enable at ingress
• Incremental (phased) rollout
• Canary vs blue/green
• Avoid datastore migrations (if possible)
24/11/2016 @danielbryantuk

34. Centralise Configuration
• Consul & consul-template
• Etcd/ZK & confd
• Netflix Archaius
• Spring Cloud config
• Watch for application rollback!
24/11/2016 @danielbryantuk

35. Operate
24/11/2016 @danielbryantuk

36. Building Blocks
• HashiCorp Terraform
– VMWare? cloud-init and vcloud-tools
– “Boot my secure government cloud”
• “CAPS”
– Chef, Ansible, Puppet, SaltStack
– Automated sysadmin
• Docker / Ami / RPM / JAR
– Machine vs Os/lang artifact
• Standardise on an OS
– Amazon Linux vs mainstream distros
24/11/2016 @danielbryantuk

37. DevOps and Programmable Infrastructure
• Devops involves programming…
• Introduce SOLID principles
• Good CI/CD principles
– Gitflow etc
– Testing
24/11/2016 @danielbryantuk

38. Service Discovery
• External
– HAProxy / nginx / ELB etc
• Client-side
– Netflix Ribbon (with Prana)
– ‘Baker Street’ (extending SmartStack)
– srv-router
• Kubernetes and CF are good to go
24/11/2016 @danielbryantuk

39. External
24/11/2016 @danielbryantuk

40. Client-side
24/11/2016 @danielbryantuk

41. Observe
24/11/2016 @danielbryantuk

42. When bad things happen, people are always involved
24/11/2016 @danielbryantuk | @oakinger

43. Monitoring and People
24/11/2016 @danielbryantuk
www.infoq.com/news/2015/06/too-big-to-fail

44. Start with the Basics
• Health checks
– Coda Hale (DropWizard) Metrics
– Spring Boot actuator
• KPIs for apps (and business)
– Assertions / invariants
– Throughput
– Queue length
24/11/2016 @danielbryantuk

45. Logging
• What every engineer should know
• 10 Tips for Proper Application Logging
• ElasticSearch-Logstash-Kibana (ELK)
– Buffer/proxy log sending or…
– Mount directory into container
24/11/2016 @danielbryantuk

46. Monitoring
• Push
– Spring Boot actuator e.g. InfluxDbExporter
• Pull
– E.g. Telegraf (shout to Tareq Abedrabbo)
• InfluxDB vs prometheus vs graphite vs
opentsdb
• Information radiators
– Aggregate vs individual
24/11/2016 @danielbryantuk

47. Aggregation: Sick Cattle, Not Sick Pets

48. Opentracing & OpenZipkin
24/11/2016 @danielbryantuk

49. Problems?
• Rob Ewaschuk’s “Philosophy on Alerting”
• Brendan Gregg’s USE method
– “check utilization, saturation, and errors.”
• “DevOps Troubleshooting”
– Kyle Rankin
24/11/2016 @danielbryantuk

50. Let’s wrap this up...
24/11/2016 @danielbryantuk

51. Summary
• development of 3+ services is challenging
• Think about your build pipeline(s)
• Testing requires a paradigm shift
• Operations requires much more automation
• Push through to prod as early as possible
24/11/2016 @danielbryantuk

52. Bedtime reading
24/11/2016 @danielbryantuk

53. THANKS...
@danielbryantuk
daniel.bryant@opencredo.com
http://muservicesweekly.com/
(Credit to Tareq Abedrabbo for inspiration/guidance)
24/11/2016 @danielbryantuk

54. Logging vs monitoring
• Logging
– Primarily used post-incident
– Machine readable
– Hard to do right
– Noise/signal
• Monitoring
– Useful in Real time during incident
– Trend analysis
– Easier to do
24/11/2016 @danielbryantuk