Mark Andersen talking about the successes and challenges of devops in Capital One Auto Finance

1. Recipe for DevOps Success:
Capital One Style

2. • Tell you about Capital One’s DevOps journey
• Describe the successes and challenges
• Give you at least 1 actionable item to take away
Goals

3. • Mark Andersen (mark.andersen2@capitalone.com)
• Alma Mater: University of Illinois – Urbana Champaign
• At Capital One for last 2 years
• Director of Devops for Capital One Auto Finance
• Capital One cloud account lead for Plano
Who am I?

4. • More than just credit cards (Auto Lending, Retail Banking, Commercial
Banking, etc.)
• Startup in bank years
• All in on devops
• All in on the public cloud
• Open sourcing some of our own products (Hygieia / Cloud Custodian)
Who is Capital One?

6. • Inconsistent releases
• Many handoffs between many teams for even the smallest change
• Manual steps documented in spreadsheet
• Environments took many months to setup and had considerable drift
Capital One Life Before DevOps

7. The Capital One DevOps Transformation
Phase1
On Prem Use of
Configuration
Management
Automation Tools
(Chef)
Automation of
Building out
Middleware and
Application Software
Phase2
Cloud Journey – Full
Infrastructure
Automation into
Delivery Pipelines
Current
Implementing Robust
Pipelines with
Increased Quality
Checks and
Automation
Pre-Approved
Releases

8. • Started with 2 legacy applications. Got a focused SWAT Team together in 1 room
for 2 months.
• SWAT Team had developers, infrastructure, and production support. Forced
cross team / cross responsibility collaboration.
• Added a few professional services resources to address knowledge gaps.
• Automated the application build, infrastructure deployment, and application
deployment.
• Tools: Jenkins, AWS CloudFormation, Chef
• Moved on to address 4 more applications of increasing complexity.
• Completed an additional 4 applications before having the application teams drive
themselves. (retired the SWAT Team)
The Capital One DevOps Transformation

9. Automation Really Does
Work and Provides Value
• When testing our first
application, we had a
bad performance test.
Changed a property to
make the servers 1
size bigger. Ran the
deployment job. 20
minutes later we were
resized and had a
successful test.
SWAT Team Cross-
Functional Collaboration
Provided Huge Dividends
• Removed many
handoffs between
teams
• Developed shared
goals for deliverables
– More “We” instead
of “They”
• People learned other
roles and learned to
respect the complexity
– i.e. App Teams
creating infrastructure
automation
Speed of Delivery Increased
• Less waiting on other
teams
• Faster feedback loop
for teams
• Test and Fail
What worked?

10. Tried to Automate
Everything all at once
• Created long pipelines
with long deploy
cycles. Also required
too many new
problems to be solved
at the same time.
Automation handoff to
application team had issues
• As automation was
returned back to
applications team, they
didn’t have some of the
training / knowledge to
grow it and support it.
Tools were setup to
support non-devops
environment
• Tools needed to be
updated to support
devops processes and
availability needs.
What didn’t work well

11. 10. Don’t just have the devops engineer know the automation. DevOps engineers need to teach
the POD how things work and how to fix things.
• The goal is for the developers to be self sufficient.
• The more the developers understand, the more then can fix their own issues without help and waiting on someone.
• Treat the automation like a car. Teach the application team to drive it, change the oil, change the windshield fluid,
etc. Not how to build the engine.
9. Prepare yourself for on prem / legacy software to be challenging to automate. You will have to
be creative.
• Cluster discovery tends to be tougher
• Consider the software you are trying to automate and deploy
Top 10 Ingredients for DevOps Success

12. 8. Don’t try to do everything at once. Focus on one or two thing first.
• If you try to do CI / CD / Environments / Cloud all at once, it will be too much.
• Smaller batches (where have I heard that?)
• Smaller chunks means you get to celebrate more. Do it
• Celebrate like Dude Perfect
Top 10 Ingredients for DevOps Success (cont.)

13. 7. If you are picking (or building) tools, make sure they have API’s.
• API’s are the enabler for devops (for cloud too)
• Even if it has a bad API, it is better than no API.
• Be API first for your tools / scripts. Build your UI after that.
6. Speed matters. Long feedback loops are bad feedback loops.
• When you automate everything the first time, you’ll find some things take a
really long time.
• Look to parallelize everything (especially functional tests)
• If you pipeline takes 1 hour to complete, are you going to wait for it for
feedback?
Top 10 Ingredients for DevOps Success (cont.)

14. 5. Unit testing is still needed.
• Don’t just focus on functional tests. They tend to be really slow and brittle.
• Unit tests are fast and stable.
• Consider mocking out your functional dependencies for speed and stability.
4. Networking and security is hard. It is even harder for developers. Training and tools are
needed.
• Centralize your log collection for everything (applications and infrastructure). Let everyone see them. Create
dashboards in “developer” language for the information they need.
Top 10 Ingredients for DevOps Success (cont.)

15. 3. Avoid creating a Devops silo to replace your
infrastructure silo.
2. Focus on removing / reducing the handoffs. Create
self service for things you can’t give developers direct
access to.
• If you can’t give developers access to create / modify them
(LDAP groups, Security Groups, etc.), give them a tool to do it
the right way
Top 10 Ingredients for DevOps Success (cont.)

16. 1. Don’t let great get in the way of being very good.
• Deliver something, get feedback, do some more.
• If you are trying to wait until it is perfect, you’ll never deliver.
• Unicorns don’t really exist (or they are tough to find). Strive to be a horse.
Top 10 Ingredients for DevOps Success (cont.)

17. • The way service teams (DBA’s, Security, Networking, etc) need to interact with agile pods is
different
• Previously, these service teams tended to optimized to their resource efficiency and their internal
workflows.
• Now, they need to shift to building products / services that are optimized to the workflow of their
agile pods.
• Understand that the service team’s customer is the agile pod’s. We need to optimize to a
solution that is most optimized to them. Not audit / security / Database leadership / etc.
Are there patterns we are seeing?

18. • DBA’s creating automation to create new database instances that can be consumed by the agile
pods by the agile pod’s running the automation themselves or calling an API to have the
automation run for them (or even a Jenkins job that the agile team has access to).
• This would create the instance, setup monitoring and alerting, etc.
• Monitoring team creating a process where you can onboard to the monitoring tool through an
API or through a config file. Process needs to be automated so it takes < 10 mins. If it a request
that is filled manually, wait time will grow to 2 weeks. A great monitoring tool with a manual /
slow onboarding process is a tool people won’t use.
• Security team having an API to create AD groups (or IAM roles) and having reference
implementation code on collecting groups within application.
Overall theme is for service teams to treat agile pods as their customer and they
should optimize whatever service / value they provide in a way that is best for the
agile pods, not the service team.
Examples

19. Thank You

20. Appendix

21. Main site Open Source Project Site = https://developer.capitalone.com/
Hygieia (DevOps visualization tool) = http://www.capitalone.io/Hygieia/getting_started.html
Cloud Custodian (cloud management) = https://developer.capitalone.com/opensource-
projects/cloud-custodian/
My email = mark.andersen2@capitalone.com
Capital One Open Source Projects