Jon is co-founder & CTO of Pitchero heading up the technology team, but currently working across the whole business to make sure they have appropriate processes in place as they grow. Jon will explain how Pitchero have used DevOps practices over the last few years to improve both technical and business agility, covering both the people challenges as well as engineering detail. http://www.leedsdevops.org.uk/post/152568213470/meetup-tuesday-15th-november-2016-at-the-odi https://twitter.com/jonmilsom https://twitter.com/PitcheroTech https://twitter.com/leedsDevops/status/798608855781490688 https://twitter.com/techdiction/status/798607947680993280

1. Increasing agility through DevOps
Jon Milsom - Co-founder & CTO
@jonmilsom
Leeds DevOps
15th November 2016

2. An online platform for amateur and semi-pro sports teams

3. Last time_
August 2014_

4. “agility”

5. “agility”
“ability of a system to rapidly respond to change by adapting
its initial stable configuration”
Wikipedia - Business agility
https://en.wikipedia.org/wiki/Business_agility

6. Examples
Things you can do if you have good agility
● Release software frequently and with confidence
● Develop new features without being bogged down by technical debt
● Onboard team members and bring them up to speed quickly
● Move systems between data centres / clouds

7. Configuration management

8. Configuration management
Containerised development environment
Aggregated logging (via Ansible)

9. Configuration management
Before Time After Time
Get a new server
into prod
Runbook hours Ansible minutes
Setup local dev
environment
- days Vagrant & Ansible minutes
Create test server
for new software
version
Runbook
Individual log files
hours Ansible
Aggregated logging
minutes

10. Configuration management - Example
PHP Versions 5.3. to 7.0

11. Configuration management - Example

12. Communication

13. Communication

14. Communication
“Business”
Marketing
Sales
Support
“Technology”
Development
Operations
Security

15. Communication
“Business”
I need it yesterday, for free
What are you working on?
We can’t do our job unless you deliver
this
“Technology”
Not possible
Last weeks last emergency project
:(

16. Jared Dunn, Silicon Valley
http://www.theyoungfolks.com/wp-content/uploads/2014/05/episode-05-05-1024.jpg

17. Communication
How to write a good bug report
Why security is important

18. Communication
Self service for other departments
Requires trust & process

19. Communication
Summary:
● Be transparent - increase visibility with other departments
● Communicate openly - good or bad
● Trust - remove unnecessary bottlenecks (self service where possible)

20. Communication
Summary:
● Be transparent - increase visibility with other departments
● Communicate openly - good or bad
● Trust - remove unnecessary bottlenecks (self service where possible)
“growing up”

21. Managing technical debt & fixing bugs

22. Managing technical debt & fixing bugs
Measure / assess

23. Managing technical debt & fixing bugs
Prioritise

24. Managing technical debt & fixing bugs
Commit

25. Managing technical debt & fixing bugs
Going forward, every sprint:
a. Fix bugs & regressions quickly
b. Refactor (retrospectively) poor decisions
c. Increase test coverage

26. Embrace open source & PaaS

27. Embrace open source & PaaS

28. Embrace open source & PaaS
RDS

29. The Database

30. The Database - why is this hard?
● Live transactional data
○ Always changing (writes), must be atomic & consistent i.e. NOT immutable!
● Size of data set
○ Hard (slow) to move around
● Changes to schema / indexes performed manually etc
○ Seen as high risk, hard to maintain across environments

31. The Database - problems we had/have
● Made changes in production, not replicated to other environments
○ Application now out of sync with database schema or indexes
● Accidently deleted data from production which was required for tests
○ Now tests fail & difficult to re-assemble required state of dataset
● Deployed schema changes with application changes & could not revert
○ Data set polluted based on application code, hard to reverse engineer

32. This might lead you to fear change...

33. “We don’t like change”

34. You can increase agility / deployability

35. The Database
https://skeltonthatcher.com/blog/database-deployability-1-minimize-changes-in-production/
Database Deployability

36. The Database
https://skeltonthatcher.com/blog/database-deployability-1-minimize-changes-in-production/
Problem: Changes originate in production

37. The Database
https://skeltonthatcher.com/blog/database-deployability-1-minimize-changes-in-production/
Solution: Isolate flow of production data

38. The Database - so far
1. Codified and version controlled changes to schema / indexes
2. Decouple tests from database contents (work in progress)
3. Deploy (backwards compatible) schema changes before code

39. Project:
“Blue Sheep”

40. Blue Sheep

41. Blue Sheep
English (UK) =>
English (USA)
English (CA)
French (CA)
French (FR)
X (Y)

42. Blue Sheep
High risk

43. Blue Sheep
Complex / Big

44. Blue Sheep
How?

45. Blue Sheep
1. “Responsive” feature toggle
Release the code without changing the UX
Users saw minimal change - we reduced risk

46. Blue Sheep
2. Re-built template by template
Continually delivered small chunks of work each sprint
Caught regressions & errors quickly

47. Blue Sheep - Tech results
● Refactored majority of public facing pages to use Laravel
● Implemented platform wide translations engine
● Converted frontend to be responsive
● Refactored CSS to use ITCSS / BEM principals

48. Blue Sheep - Business results
● Pages per session up 51%
● Session duration up 97%
● Bounce rate down 37%
● Marketing team now able to send out location specific campaigns

49. We were on TV :)

50. We were on TV :(

55. Ego vs team

56. Your ego has its place

57. Often ego win = team fail

58. We want the team to win...

59. Personal examples
Me Team
Aim to work with (or hire) people
who are better than you / have
other skills
No longer “the best” Skillset increased & augmented
Everyone learns
Had a pull request rejected Feel like a n00b! Standards have increased
Trust people to make big
decisions
Scary to delegate Shared domain knowledge
Trusted & empowered

61. “Leave your ego at the door”
#egoless technology teams
https://www.allaboutjazz.com/an-evening-with-a-legend-quincy-jones-quincy-jones-by-solomon-leflore.php

62. The Future
Codify entire infrastructure (currently just individual servers)
Continue to invest in reduction of tech debt
Remove noise from error logs
Consolidate monitoring & analytics
Increase quality and quantity of automated testing
Parallelise CI

63. @PitcheroTech / @jonmilsom
Questions?
We are hiring! pitchero.com/careers