Infrastructure is not the only outcome. ThoughtWorkers Will Garcia and Pete Oxenham share their journey to production with Azure including practical tips on how to introduce incremental change, improve developer experience and enable flexible engineering practices in autonomous teams

1. CUSTOMER CENTRIC DELIVERY TO THE CLOUD
William Garcia & Peter Oxenham

2. 2
Ready to
make your
mark?
/careers
©ThoughtWorks 2019 Commercial in Confidence

3. ©ThoughtWorks 2019 Commercial in Confidence
What we found
Separate Dev & Infrastructure Teams

4. ©ThoughtWorks 2019 Commercial in Confidence
What we found
Dedicated Infra Team Member

5. ©ThoughtWorks 2019 Commercial in Confidence
What eventually got to
Product Team

6. ©ThoughtWorks 2019 Commercial in Confidence
Challenging debuts
Our cards initially

7. ©ThoughtWorks 2019 Commercial in Confidence
Why “tech work” creates chaos?
Scope creep?
Impact / user
value?
Size?
Taking too long?
De-prioritisation
Accepted risk
Tech debt wall
Manual Checklist
LIMBO ZONE INTERPRETATIONS
Back and forth process
Low self-esteem /
demotivation
Capacity issue
Acceptance of Knowns
issues
Need to put more
control in place
Wait and see
RESPONSECONFUSION
ZONE
U turn
And more

8. ©ThoughtWorks 2019 Commercial in Confidence
The customer’s angle
Our new format

9. ©ThoughtWorks 2019 Commercial in Confidence
Example - Disruptive tenants
Do not cause application slowdowns
9
Poor performance isolation across
customers
BEFORE
Customer perception of reliability
Automated actions on new
performance patterns
Define SLOs
Review Azure Service Limits &
Autoscaling rules
Graph metrics in the team dashboard
Traffic is routed based on response
times
Performance trends are visible to
anyone
BEFORE AFTER
STEPS
OUTCOMES

10. ©ThoughtWorks 2019 Commercial in Confidence
Example - Developers don’t wait
For long builds
10
Build + Test time 40m
Caching of packages (Nuget, NPM,
Docker images)
Always-on CI agents
Build + Test time 15m
Builds run in parallel
Artefacts are cached
BEFORE
Deliver at a faster pace
Less context switching
Improved feedback loop and
error detection
AFTER
STEPS
OUTCOMES

11. ©ThoughtWorks 2019 Commercial in Confidence
Find the outcomes
Simple approach to infrastructure / CI-CD delivery
Customer
outcomes
Dev Experience /
Efficiency
Improved user
experience
Reduced internal/
customer effort
Enhanced
customer trust
Target
State
Team knows which
version is deployed
Developers can use any
tool they need in the CI
system
QA don’t wait for
environments
Client onboarding is
easy
Multi-tenant: each
client has its own
access URL
Reduce support
activities
Reduce bottlenecks
during peak hours
Proactively find and fix
users issues
No service interruption
during business hours
Updates are frequent
Reduced MTTR
Data access is “silo-ed”
and protected per client
11

12. ©ThoughtWorks 2019 Commercial in Confidence
Unfold the fundamentals
Initiatives
✓ Quick updates
✓ CI / Environments
workflow
✓ Versions endpoints
Self-service databases
Azure App Service App
Insights
Azure Monitor & Health
Azure resources tagging
Client-side to server-side
logging
HTTP Health checks
Product and feature metrics
Azure App Service slots
Azure ASE
Application Gateway / WAF
IRP template
Hawkeye/RetireJS/OWASP
scanners
Tools/
Techniques
Azure App Service Perf
testing (user
load/duration)
Terraform
JMeter / Gatling /
GOReplay
Azure CDN
Azure Advisor
Performance testing
✓ Autoscaling
✓ Reliability
Circuit breaker
✓ Disaster recovery
✓ Blue/Green
Canary deployments
✓ Incident response plan
✓ Security in the pipeline
✓ External monitoring
✓ Logging and correlation
✓ Cloud cost analysis
✓ Product dashboard
Subdomain delegation
Pipeline as code pattern
Docker
Dev Experience /
Efficiency
Customer
outcomes
Improved user
experience
Reduced internal /
customer effort
Enhanced
customer trust
12
Simple approach to infrastructure / CI-CD delivery

13. ©ThoughtWorks 2019 Commercial in Confidence
But Why?
What are the benefits?
● Lean Approach
● Incremental Change
● Change is embraced
© Copyright, Lean Enterprise Institute (www.lean.org). Used with permission.

14. 14
©ThoughtWorks 2019 Commercial in Confidence
Engineering principles applied to
infrastructure on Azure

15. ©ThoughtWorks 2019 Commercial in Confidence
Responsibility for full cycle of
application
Bring automation and engineering
rigor to the full application cycle
Be willing to change the current
process
Automated testing and deployments
Monitoring and logging in Azure
Docker to speed things up
Security pushed to the left
TEAM HOW WHY
Optimize for getting value-
increasing changes into production
quickly
Create stability / reliability
Remove waste
+ + + +

16. ©ThoughtWorks 2019 Commercial in Confidence
Pipeline as code vs. DevOps tasks
16

17. ©ThoughtWorks 2019 Commercial in Confidence
Improved dev experience
$ make app-ui-test
$ make app-backend-test
$ make app-build
$ make app-e2e-test
For build, test, troubleshooting, deploy, security, onboarding
$ make monitor-logs
$ make monitor-health
$ make security-deps-scan
$ make security-secrets-scan
$ make deploy
$ make deploy-test
$ make backup
$ make undeploy
17

18. ©ThoughtWorks 2019 Commercial in Confidence 18
Run on “any CI agent, any software”
With a Docker only approach

19. ©ThoughtWorks 2019 Commercial in Confidence
Our path to containerisation
Start with low risk, high value tasks
Docker on
Windows
Linux containers
on Windows
Docker compose
Switch to Linux
Azure Container
instances
Azure Kubernetes
Service
Dev databases App. CI Builds & Tests Deploy Security APIs
Everything now runs in
Docker

20. ©ThoughtWorks 2019 Commercial in Confidence
Rapid infrastructure prototyping
With AzureRM generated templates
● “static” infrastructure
● Experimental infrastructure
● Development services

21. ©ThoughtWorks 2019 Commercial in Confidence 21
Declarative Infra. As Code
With Terraform and Terraform AzureRM provider

22. ©ThoughtWorks 2019 Commercial in Confidence
Infrastructure stacks
Loosely coupled
22
Modular infrastructure
● Evolvability
○ Architecture
○ CI Pipelines
● Failure isolation
● Reusability
● Shorten deploy / undeploy times
● Tests stay closer to what is being deployed
○ think unit tests

23. ©ThoughtWorks 2019 Commercial in Confidence 23
Infrastructure testing
To save the day!

24. ©ThoughtWorks 2019 Commercial in Confidence 24
Security and compliance testing
Network and security policies are enforced at deploy time

25. ©ThoughtWorks 2019 Commercial in Confidence 25
Infrastructure smoke testing
Pinging the health checks

26. ©ThoughtWorks 2019 Commercial in Confidence
Beyond pinging the health checks
/info
/health
Telemetry, degradation rules, performance, dynamic routing

27. ©ThoughtWorks 2019 Commercial in Confidence 27
Notifications as a communication tool
For development and production
Azure App Insights is down New Docker image available

28. ©ThoughtWorks 2019 Commercial in Confidence 28
Server and client side monitoring
With Azure Application Insights

29. ©ThoughtWorks 2019 Commercial in Confidence 29
Setup for Success
Continuous Delivery
● Deployment frequency - 79 PROD releases in June
● Lead time (code commit to PROD) - 55 minutes
● Mean Time To Recovery - ~30 minutes
● Service Availability - 99.864%

30. TAKEAWAYS
● Continuously learn and assess the fundamentals
● Aim for customer outcomes based infrastructure
● Adopt an iterative approach to building
capabilities
● Have fun with tech and most importantly with
the people!

31. 31
Questions?

32. ©ThoughtWorks 2019 Commercial in Confidence
Thank you
32
Will Garcia & Peter Oxhenam