Matt Reynolds discusses how building internal tools can help development teams become more self-sufficient and reduce bottlenecks. Some key points are: 1) As development teams gain more autonomy, traditional operations teams are overwhelmed, so tools can help developers account for operational concerns from the start. 2) Tools can standardize approaches, reduce specialist knowledge requirements per team, and speed up common operations like provisioning resources and testing environments. 3) Internal tools reduce friction between teams and flows, allowing operations specialists to take on more impactful work or reduce infrastructure spend. Regular feedback ensures tools address real user needs.

1. Paving The Road To
Production
Why you need an internal tools team
Matt Reynolds
@mattreyuk
Hi my name is Matt Reynolds I’m a senior engineer in the data and infrastructure
group at Ibotta a rewarded shopping app built here in Denver. My talk today is about
how building tools can help make your life - dev or ops a little easier.

2. Observe
Orient
Decide
Act
First - the problem. Like the fighter pilots this theory was originally developed for,
companies are trying to get round this loop faster and faster and learn from the
changes they introduce to their customer base before their competitors can match.

3. Autonomous Teams
To go faster, they’re also introducing more independent teams with more autonomy to
produce their systems - breaking the old release trains and deploying independently
at a pace that works for the individual team

4. What About Ops?
The traditional Ops group at the end of the release path was already in trouble but
now Product Managers are driving functional requirements in all these teams, there
are rarely enough ops to embed in each dev team to inform on non-functional
requirements.

5. What Can We Do?
To overcome this, we need to help developers become more self sufficient - ensure
that as much as possible, operational concerns are built in from the ground up by
working with development to pave a road to production with tools.

6. How do I Start a
Project?
There are opportunities to provide tooling throughout the lifecycle of a service starting
with service template generators to get you going faster and help standardize
approaches. Yeoman is a great generator tool - there’s even a yeoman generator
generator

7. How do I Provision
Resources?
Using something like terraform template modules is a great starting point to build on
in helping devs provision their own resources but you’ll also want to build guardrails
using alerts for cost overrun and security issues so you can “trust but verify”

8. How do I Make it
Easier to Support?
Building standard configuration and libraries for your supported languages gives
developers an easier way to include things like request tracing, structured logging and
monitoring, configuration and secrets management

9. How do I run it in
Dev/Stage/Prod?
Docker can make it much easier for devs to test locally, using compose for
coordinating several containers. Then mocking, stubbing, contract testing together
with controlled testing in production can reduce the need for large staging
environments and the overhead they entail

10. How do I Manage it in
Production?
Based on the conventions we’ve established with earlier tools we can now automate
things like connecting to log aggregators, building standard dashboards and alerts
and a common way to deploy or roll back changes. Making “you build it, you own it”
more of a reality

11. Benefits:
Standardization,
Speed
Building these tools helps us to standardize our approach - it encourages common
patterns and allows people to more easily move between teams and projects, reduces
the need for specialist knowledge per team and the time required for common
operations.

12. Benefits:
Flow,
New Capabilities
This should reduce bottlenecks waiting for access to those specialists and allow them
to do more impactful work (like writing even more tools!) or maybe working on
reducing infrastructure spend, chaos testing for resilience etc

13. Starting Top Down
If there’s a big new initiative (and let’s face it, there usually is) - you may be able to
drive a top-down change by advocating how internal tools can help the company
achieve that goal - just remember to build a case on the business benefit, not the
technology details

14. Starting Bottom Up
… but you also need to build from the bottom up - often you’ll want some concrete
example you can point to to strengthen your case so find a pain point - especially one
between teams and build a tool to help reduce that friction

15. Write the Right Thing
You want to build things that will actually get used - so find early adopters prepared
for some pain, get them trying early rough versions, iterate on their feedback before
you open up to wider audience. Be sure to focus on the main use cases - the old
80/20 rule.

16. Listen to Your
Customers
Along with that, you need to really listen to the users of your tools, empathize and
understand their pain points - be open to their feedback - it won’t always be nice to
hear but try and find the underlying issue that’s causing the frustration so you can do
something about it

17. Keep it Going
Congratulations! You’ve created a bunch of tools, now how do you keep them
relevant and up to date? Try adopting an internal open source approach, accept PRs
on tools from users and then help them to take ownership when they’re mature

18. Support
Make use of your early adopters as evangelists, they can publicize and help with low
level support questions. Documentation is also really important as it helps provide a
consistent message.How-tos and troubleshooting guides can help with self service
but you still need an open support channel.

19. Expand to Other
Areas
internal tools doesn’t just have to be about dev and ops - you can expand the
principle to other groups in the organization such as data engineering, analytics and
business support teams - maybe even integrate with IT for easier on-boarding etc

20. Conclusions
Hopefully you’ve seen that by developing your own tools, you can help alleviate
pressure on operations specialists, empower developers to own more of the process
and improve the operational readiness of your services