Join us at Agile+DevOps East's DevSecOps Summit on November 18th to check out our new presentation: https://agiledevopseast.techwell.com/program/devsecops-summit-sessions/blameless-retrospectives-devsecops-global-healthcare-giants-agile-devops-virtual-2020

1. Blameless
Retrospectives in
DevSecOps
(At Global Healthcare Giants)

2. @aaronrinehart @verica_io #chaosengineering
Aaron RinehartDossier
Aaron “The Kaiser” Rinehart
a former Chief Security
Architect at United
Healthcare, and formerly a
renegade at DoD and NASA is
a frequent speaker and
author on Chaos
Engineering. As a pioneer
behind Security Chaos
Engineering he’s authored
Chaos Engineering, and
Security Chaos Engineering
books for O’Reilly.

3. Dossier
DJ “The Mad Professor”
Schleen, a DevSecOps
pioneer and Application
Security Manager at Rally
Health tried to evade
capture while boarding a
steam liner with crafty
automation tactics. While
on the run for the last 10
years, he’s been involved
with the “DevOps Crew” and
has been preaching about
automated security at
gatherings across the
world and in the books
that he has written.
3@djschleen @rally_health #deadpool

4. Struggle

5. Empathy

6. The Journey

7. The Journey
It all started when I was part
of a startup recently acquired
by a massive healthcare
organization.
We needed to rapidly address
problems with new technologies
and built our own orchestration
when no other tools were
available
We started with nothing and
needed to steer the boat
towards AppSec as a practice.
Security had become more than
GRC, TPRG, IAM, SOC.
There were pockets of
knowledge, no centers of
excellence in software
security.
A Grass Roots Beginning
• Teams across Silos & Disciplines w/
No Funding
• 60 Developers, Operations
Engineers, and Security Leaders
from across the entire company.
• Began with Six Core DevOps Security
Problem Sets
• Security Baseline + Configuration
Validation w/ Chef & Inspec
• Gauntlt Rugged Attack Framework
• Static Code Analysis (SAST):
Automating Fortify with Jenkins via
API
• Application Vulnerability
Scans(DAST): Automating WebInspect
with Jenkins via API
• DevOps Self-Governance &
Operationalization Framework: How
does this world look from an
operational support perspective?
• Clair Container Image Scanning:
Building Image Scanning into
Jenkins

8. The Good Parts
Reduced overall
vulnerabilities in our code
base
Educational programs
(mentoring, champions, etc)
helped both developers and
security engineers understand
the challenges facing each
other
Codifying automation improved
efficiency
Developers could react to
vulnerabilities and zero days
faster than they could
without security in the
mindset
We Didn’t Sink
Successfully delivered an open
source DAST tool into CI/CD
pipeline to driveinstrumentation of runtime
security left in the delivery
pipeline
Drove down delivery times of
highly regulated workloads by
automating the verification of
security hardening configuration
using InSpec
Delivered the ability for teams
to initiate their own DAST/SAST
scanning via API
Adopted Commercial IASTsolution. Took a very long time
to procure but saved the company
millions of dollars inefficiency per month.
Built empathy within theSecurity Organization byadopting a Everyone Must Learn
to Code learning development

9. A hole in the wash basin.
COLLIE SHANGLES! We
started with a SAST
program first - should
have started with OSS
Started with tools but
should have started with
relationships
We foolishly looked at
integration first before
knowing where the
highest risk application
code was!
Can we have a do-over?Focus on top downtransformation more.Bottom-Up was moresuccessful until we hit a
point of needing fundingto go further
Spend more time helping to
transform flagship company
products. This sets theproper tone for the restof the enterprise.
Spend more time educating
security counterparts onthe business value of what
transforming

10. Lessons

12. No Lifeboats!
We failed builds basedon security
vulnerabilities beforewe helped burn downvulnerabilities.
Doing this blocked
production deployments.
Blocked deploymentsmeant controls weretaken out without
security knowledge.
We tried to move tooquickly and didn’t planas much as we shouldhave
A Compass Please
Initially automating our
existing SAST/DAST
scanning tools via API
caused the scanning
infrastructure to crash.
The servers that supported
it could not withstand the
volume.
Initially implementing
Secrets Management was
difficult. Security teams
did not understand what
software secrets were.
There was confusion
between Secrets and
Privileged Accounts.
Docker Container Image
Scanning with Clair didn’t
meet needs

13. All Aboard?
If people aren’t on board,
nobody cares.
You’re dealing with traditional
security organizations being
assholes. That shouldn’t be
surprising.
Don’t invite people to your
party if you aren’t ready yet.
Know (or at least have a good
idea) where the highest risks
are.
Look before you automate (look
before you cross the road)
It's the human fear of not
being in control that hinders
automation.
Learn to Navigate!
Important Skills are Listening and
Mutual Empathy
Show something Built is Better than
an Idea
Fail small, fail fast
Its a culture shift, not just about
automation
Continuous Learning is more
important than Continuous Fixing
Don't try to reduce complexity,
learn to navigate it.
Avoid Analysis Paralysis: DevOps is
a culture and a living organism
DevOps is not a fad, it is the
future
Automation is Important but “Don’t
be Distracted

14. Predictions

15. Malarkey.
Tools haven’t caught up yet. We
are using flashlights for high
mounted brake lights and
feathers when we need airbags
Current security tooling won’t
suck
We’ll see more innovation in
the detection of security
issues
Tighter feedback loops for
security issues - fixing
security issues with confidence
of break risk
In five years DevSecOps becomes
known as “Engineering”
Witchcraft.
The Next Generation of Security
Professionals will be Chosen
from DevOps Teams
Shared Responsibility becomes
more of a reality.
Security continues the move
towards value
stream
Security becomes a recognized
skill within Site Reliability
Engineering (SRE)
Chaos Engineering becomes a core
discipline within DevSecOps
Compliance in DevSecOps becomes
a byproduct of good engineering
practices

16. The End.

17. @djschleen @aaronrinehart
cutt.ly/verica-book
Free copy mailed to you complements of Verica
Blameless
Retrospectives in
DevSecOps
(At Global Healthcare Giants)